For years now I've been told to start documenting some of the ridiculous things I do. I tend to do things incorrectly, making them unnecessarily complicated in order to work with an existing system.
Effectively, I like retrofitting. Completely ridiculous things like making Windows 7 silently RDP to itself in order to log in multiple users on boot is unheard of. Why? Because we may as well use Windows Server, but that's no fun. That's the kind of ridiculous project I embark on.
But today, something a little more useful.
Note that this is not really a tutorial, more a journal following my progress on this project, as such there will be back-tracking and random commands that may not actually be useful in the final product. It's advisable to read through and modify it into a tutorial if you'd like it to be efficient.
Cloud Storage, In Ways It Was Not Meant To Be Used
Everybody has heard of and probably used cloud storage. Dropbox, Mega, SkyDrive, whatever. Very recently I found out about Amazon's contender: Amazon Cloud. I hadn't heard of it before, probably because it skimmed its way under the radar here in the UK as it doesn't offer anything particularly crazy. In the US however, they offer an "Unlimited Everything" package, unlimited file storage for only $60 a year.
As a person with near enough 5TB of media and on a network that doesn't allow port forwarding (Student life!), this caused a pretty severely raised eyebrow. If I could store and access all of my media on the go from a reliable source, not only could I cut down on my drives at home (All of which are fast approaching "Warning" on SMART due to 20k+ hours of uptime), but I would be significantly more comfortable on trains, especially with the PlexSync(assuming I ever actually bother getting a PlexPass). Not to mention it means I could just set up my dedicated box (which is already a torrent server) to upload straight to Amazon and have it accessible to all of my devices.
So let's go through the steps of trying to get a cloud based media server running.
Requirements
An Amazon account: Thankfully, if you're not willing to shell out $60 for an unlimited everything package just yet, Amazon offers 5GB for free for you to play with before you upgrade, so I'll be using that. This is going to be our cloud storage.
A Dedicated Hosted Server: Yeah, this is where most people fall short. I happen to have one of these sitting in France from Kimsufi for £3 a month which I use for hosting a few websites and services. Any VPS or server will do as long as you can forward ports and it has a decent download and upload speed. This is going to be the "head" for our storage: We'll trick it into thinking the Amazon storage is local, then use it to stream data to our clients. Effectively, it's a proxy.
Some Linux Knowledge: For this project I'm assuming the dedicated server is an Ubuntu based server. In theory it should work fine with any Ubuntu based distribution, and with some modifications of the programs in use it could even work with Windows.
A PLEX account: PLEX is the easiest way I've found to quickly stream media to pretty much any platform, so I'll be using that to try and view my library online.
Where do we start?
First of all we really should get some test data on our Amazon Cloud. I used a YouTube music video I had sitting around. You can start uploading your entire media library if you like, but for now I'm being conservative. On a desktop machine, uploading single files is trivial, merely go to your Amazon Cloud Dashboard and drag a video file or two onto the browser window.
Next, we should make our media visible on our dedicated server. For this we'll be using an open source app called acd_cli by a user on GitHub called yadayada. It's available for viewing here. Effectively, it's a command line interface that symlinks your cloud drive to an accessible part of the filesystem, so you have read/write access.
SSH into your server and get ready for some terminal action. Due to some issues with the latest version of pip, we actually have to grab easy install(from python-setuptools) and a bunch of other dependencies then use that to install pip(the python package manager).
sudo apt-get install python-setuptools python3-appdirs python3-dateutil python3-requests python3-sqlalchemy
Then pip:
sudo easy_install3 -U pip
Then we'll be wanting to grab acd-cli and install it from source:
pip3 install --upgrade git+https://github.com/yadayada/acd_cli.git
Once that's installed, we can auth ourselves.
acd_cli sync
Now you've been given a text browser to work with! It might actually be easier here to navigate on a machine with a GUI to get your key, but if you want to use text, good luck. I used my PC for this step.
Navigate to https://tensile-runway-92512.appspot.com/ , which will direct you to Amazon to login. This step is to authorise the application to your Amazon account.
After authorising, it should have downloaded a file called "oauth_data" for you. SFTP into your dedicated server and drop this file into /home/username/.cache/acd_cli/
You can now go back to your server and press Q to quit out of the text browser. Then, press a key to continue. Acd should quickly sync. We may actually have to set up a cron to do this for us later since it's required to see new files.
For now, make a folder in your home folder, then mount your cloud drive.
mkdir amazon
sudo acd_cli mount amazon
At this point, we can cd into amazon to see our files as read only, wohoo!
At the moment, we have to manually run acd_cli to see new files, but we can set up a cron later for that.
Getting Plex Installed
So now we have a working read-only folder full of our test media. Let's install Plex and see if we can make it host it. Let's cd all the way back into our home directory and grab plex.
cd ~
wget https://downloads.plex.tv/plex-media-server/0.9.12.4.1192-9a47d21/plexmediaserver_0.9.12.4.1192-9a47d21_amd64.deb
After it's finished downloading, let's install it via dpkg
sudo dpkg -i https://downloads.plex.tv/plex-media-server/0.9.12.4.1192-9a47d21/plexmediaserver_0.9.12.4.1192-9a47d21_amd64.deb
Now, if we head to http://serveraddress:32400/web , we get the Plex login! Nice and easy!
Login to Plex and we'll be able to see the client side of Plex. Unfortunately at the moment we can't actually configure Plex since we're a client, so this is where it gets complicated again.
We have to set up a tunnel so Plex thinks we're local.
I'm assuming you've been using Putty to connect to your server via SSH the whole time. If you haven't, you'll have to find another guide on opening a SSH tunnel on your platform. This also assumes you're not running Plex locally on your machine.
Open up Putty, enter all your normal connection settings, then go down to the SSH menu, and open Tunnels. In source port, enter "32400" and in destination enter "localhost:32400". Once you login, you should have a working tunnel.
In your browser, navigate to http://localhost:32400/web . You may need to log in again, but once you do you can click the settings button in the top right. Set it up how you like. You may need to add your login once more, You may also want to tick "Update my library automatically".
Once done, hit the home button at the top left and find your server in the navigation pane on the left. Click the little + button to add a new library. In my case, I knew my test video was formatted weirdly so I picked Home Video, and added Amazon as a library.
But oh no! We can't see any media! That's fine, it's because your Plex account doesn't have permission for that folder. We'll need to remount that folder as the Plex account, since it won't allow us to chmod(change permissions) on that folder.
So first, let's unmount it.
sudo fusermount -u /home/username/amazon
Interesting things now need to happen again. Since the mount is locked to whatever user made it, and the we cannot change read permissions of it (even as root!), we have to do everything from the Plex account. Since it's a service account that we don't know the password of, we'll need to be root for a bit, just to make things simpler.
sudo su
Now we need to change the password for the plex user account. This could be a problem later, but we'll deal with it when we get there.
passwd plex
sudo fusermount -u /home/username/amazon
Interesting things now need to happen again. Since the mount is locked to whatever user made it, and the we cannot change read permissions of it (even as root!), we have to do everything from the Plex account. Since it's a service account that we don't know the password of, we'll need to be root for a bit, just to make things simpler.
sudo su
Now we need to change the password for the plex user account. This could be a problem later, but we'll deal with it when we get there.
passwd plex
Enter your new password. We also need to make Plex a sudoer (an admin) so we can properly set things up as them.
adduser plex sudo
Now this could be considered a pretty bad idea, since now you have two accounts (whatever your other account is) and the Plex user as users you can SSH in as. If you want to change that, we can disable ssh login login as the plex user like this:
nano /etc/ssh/sshd_config
Scroll down to the bottom and type
DenyUsers plex
Save and exit, then run
Finally, let's become the Plex user.
su plex
The problem now is that if we try and run acd_cli it's going to whine that it can't find it, that's because it's looking for it in our home directory when it's in fact in your other users home directory. Can't see it? It's a hidden folder called .cache
Instead of copying over cache (I tried that, it doesn't work), we're actually just going to install all the dependencies for the plex user using sudo's -H flag, which sets the users home directory rather than the root directory.
sudo -H pip3 install --upgrade git+https://github.com/yadayada/acd_cli.git
We should now have a working acd_cli install for plex. Knowing that, we can go through the process of authenticating ourself again. Thankfully, we don't need to do the entire thing again since we already have the oauth file, and we can just copy it over into plex's SERVICE directory (NOT the home directory!). We also need to fix the permissions of it after we copy.
sudo cp /home/username/.cache/acd_cli/oauth_data /var/lib/plexmediaserver/.cache/acd_cli/
sudo chown plex /var/lib/plexmediaserver/.cache/acd_cli/oauth_data
acd_cli sync
acd_cli should then sync, and we can start setting up our folder again.
Now you may be asking why we have acd_cli on two users now. Well, the plex user is for plex, and I'd like to keep it that way. The other user will hopefully be the user where the rest of the stuff will eventually happen, like file syncing. Due to the permissions limitation we may actually have to have multiple mounted drives, but we'll deal with that later.
Since we hadn't really bothered yet, let's make the plex home directory owned by plex.
sudo chown plex /home/plex
And then let's make a directory for the mount and mount it.
mkdir amazon
acd_cli mount /home/plex/amazon/
Here I had the odd error that I didn't have permission to open a couple of files, mostly because I don't, but it seems that it mounts it fine anyway. The problem being if you run it as root (or a sudoer), the permission of the folder is for root and you can't access it.
Finally, go back onto your PC and get the plex browser tab back up. Add a new library on the left (And delete your old one, if it's broken), putting in the location as /home/plex/amazon/ .
Pick a piece of media to stream. If it works, congratulations!
In my case, I've confirmed it's able to play a 40mb video. I'm really not sure how the mounting works and if Plex has to copy the file to a cache before it will play or if it will be fine with the probably linear download. It might hiccup with large videos, but I haven't had a chance to test it yet.
You now, in theory, have a cloud base media storing and sharing service. You should be able to watch your videos from any browser or device now, while having none of it stored locally. Wohoo!
NOTES:
For now, if you add anything new to the cloud drive you'll have to "su plex" over to the plex account and run "acd_cli sync".
Similarly, if you restart you'll need to "su plex" over to the plex account and mount the drive again.
If you're having long loading times when starting videos or the video quality is a bit low, make sure you head into your Plex settings, go to Player and set the remote quality to "Original". This will use more bandwidth but will not attempt to live transcode.
Additions:
A decent size file was added, weighing in at approximately 800mb. It took about a minute to start streaming but ran fine the whole way through once it started. I'm not sure what causes the minute long delay, it COULD be the server starting to transcode it, as it doesn't seem to be caching it anywhere. I'll look into it when I can.
TO DO:
A cron for auto-syncing.
Set up a torrent server with a plugin that automatically uploads the files.
Mount on boot
adduser plex sudo
Now this could be considered a pretty bad idea, since now you have two accounts (whatever your other account is) and the Plex user as users you can SSH in as. If you want to change that, we can disable ssh login login as the plex user like this:
nano /etc/ssh/sshd_config
Scroll down to the bottom and type
DenyUsers plex
Save and exit, then run
Finally, let's become the Plex user.
su plex
The problem now is that if we try and run acd_cli it's going to whine that it can't find it, that's because it's looking for it in our home directory when it's in fact in your other users home directory. Can't see it? It's a hidden folder called .cache
Instead of copying over cache (I tried that, it doesn't work), we're actually just going to install all the dependencies for the plex user using sudo's -H flag, which sets the users home directory rather than the root directory.
sudo -H pip3 install --upgrade git+https://github.com/yadayada/acd_cli.git
We should now have a working acd_cli install for plex. Knowing that, we can go through the process of authenticating ourself again. Thankfully, we don't need to do the entire thing again since we already have the oauth file, and we can just copy it over into plex's SERVICE directory (NOT the home directory!). We also need to fix the permissions of it after we copy.
sudo cp /home/username/.cache/acd_cli/oauth_data /var/lib/plexmediaserver/.cache/acd_cli/
sudo chown plex /var/lib/plexmediaserver/.cache/acd_cli/oauth_data
acd_cli sync
acd_cli should then sync, and we can start setting up our folder again.
Now you may be asking why we have acd_cli on two users now. Well, the plex user is for plex, and I'd like to keep it that way. The other user will hopefully be the user where the rest of the stuff will eventually happen, like file syncing. Due to the permissions limitation we may actually have to have multiple mounted drives, but we'll deal with that later.
Since we hadn't really bothered yet, let's make the plex home directory owned by plex.
sudo chown plex /home/plex
And then let's make a directory for the mount and mount it.
mkdir amazon
acd_cli mount /home/plex/amazon/
Here I had the odd error that I didn't have permission to open a couple of files, mostly because I don't, but it seems that it mounts it fine anyway. The problem being if you run it as root (or a sudoer), the permission of the folder is for root and you can't access it.
Finally, go back onto your PC and get the plex browser tab back up. Add a new library on the left (And delete your old one, if it's broken), putting in the location as /home/plex/amazon/ .
Pick a piece of media to stream. If it works, congratulations!
In my case, I've confirmed it's able to play a 40mb video. I'm really not sure how the mounting works and if Plex has to copy the file to a cache before it will play or if it will be fine with the probably linear download. It might hiccup with large videos, but I haven't had a chance to test it yet.
You now, in theory, have a cloud base media storing and sharing service. You should be able to watch your videos from any browser or device now, while having none of it stored locally. Wohoo!
NOTES:
For now, if you add anything new to the cloud drive you'll have to "su plex" over to the plex account and run "acd_cli sync".
Similarly, if you restart you'll need to "su plex" over to the plex account and mount the drive again.
If you're having long loading times when starting videos or the video quality is a bit low, make sure you head into your Plex settings, go to Player and set the remote quality to "Original". This will use more bandwidth but will not attempt to live transcode.
Additions:
A decent size file was added, weighing in at approximately 800mb. It took about a minute to start streaming but ran fine the whole way through once it started. I'm not sure what causes the minute long delay, it COULD be the server starting to transcode it, as it doesn't seem to be caching it anywhere. I'll look into it when I can.
TO DO:
A cron for auto-syncing.
Set up a torrent server with a plugin that automatically uploads the files.
Mount on boot
This is the first kind of ridiculous project that I've posted, so thanks for sticking with me for it. Hopefully you'll find it useful.
Greetings!
ReplyDelete