Recovering Greyhole After Ubuntu Upgrade

In my previous post I talked about some issues I hit when I upgraded from Ubuntu 14.04 to 16.04, it wasn’t all plain sailing and in this one I’ll cover the issues I’ve had with getting Greyhole back up and running.

At the end of the last post I had my “missing” disks mounted and I mentioned I was moving data around.  Thankfully the two disks that were mounting fine we’re two of my largest, 4TB and 2TB worth, the two that weren’t mounting are 2TB and 3TB, after deleting a load of old files and reducing the redundancy level on the non-critical shares it looked like I’d have just enough space to make things easier.

One at a time I ran the command to remove a disk from the pool and waited for Greyhole to finish balancing;
greyhole –going=/mnt/three/gh

You can see what the Greyhole service is doing by running “greyhole –L”, one it tells you it is sleeping you can crack on with the second disk.

This completed and I was able to see my files from a remote machine via Samba, huzzah!  The problem was the install wasn’t tidy any more, I couldn’t control Greyhole using the service command and the landing zones were on a disk I was intending to reformat.  I tried unsuccessfully to fix it but decided to follow the steps to reinstall it in the end.  From the perspective of the documentation this would be the same as migrating to a new machine.

First off I ran “sudo apt remove greyhole –purge” which removes the service with extreme prejudice and I then followed the standard steps to install as per this page.  I restarted Samba and Greyhole after running the fsck command and lo and behold I got most my shares back online!  Two were showing up fine, full of files, one was showing up but empty.  This was my backups share which was a little worrying but I’d already backed it up to another machine so wouldn’t be a big issue to rebuild it.

It turns out that when I was configuring the smb.conf and greyhole.conf files I called the backup share “Backups” rather than “Backup” and this meant that Greyhole couldn’t find the files to make them accessible again.  I fixed this typo, ran fsck again and they are now showing up.

Regarding the other two drives, it looks like I’d initialised them as zfs_members at some point and with Ubuntu 16.04 and they can’t be mounted in the same way.  It’s a vaguely educated guess so happy to be corrected!  To get rid of them I used the wipefs tool which strips the drive bare of partition signatures.  BE VERY CAREFUL WITH THIS!

I ran “wipefs –all /dev/sdc” and “wipefs –all /dev/sdd” which seemed to do the trick.  After that I followed this guide to format my drives using parted.  I’ve no idea why but blkid still doesn’t show the UUID for the partitions I’d created but I took note of them from the output of the mkfs.ext4 command.  I put them into fstab along with creating a folder to mount against with the other two drives and ran “sudo mount /dev/sdc1” and the same for sdd1, they then showed up!

Finally I added the two drives to the Greyhole storage pool by following this guide and ran “greyhole –balance”.

A massive faff but a great learning experience!

Ubuntu 16.04 Upgrade, What Could Possibly Go Wrong?

So I logged in to my home server recently and found in the MOTD that an upgrade from 14.04 to 16.04 was available.  Being a bit cautious about things I asked a colleague if he’d done the upgrade and he had, the only issue he’d come across was for hardware I don’t use so thought I’d crack on.

That night I got home, ran do-release-upgrade, answered a few questions and left it to it.  It carried on tinkering with one of my programming projects on my desktop PC and several hours later, tired after a satisfying nights hacking, I shutdown my desktop.  Completely forgetting I had an SSH session open…
I promptly logged back on and checked my server, in HTOP there was a process at the top of the list that looked upgrade related so I left it to it overnight.  Turns out that as I didn’t have screen installed there was no way to reconnect to that upgrade session which was an arse to say the least!  I didn’t have a choice, that I know of, but to reboot.  I did so and it kernel panicked on boot, something to do with not being able to mount something.

image
BORK BORK BORK BORK ::PANIC:: BORK BORK

::Expletive deleted::

I loaded in to maintenance mode by selecting the advanced option on reboot and looked at what was or wasn’t mounted.  It turns out two of my four disks weren’t being mounted by fstab on boot, I ran blkid and they weren’t listed either.  I managed to find the following command on Ask Ubuntu which showed that the disks we’re still being detected which was a good sign.

sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL

image

I managed to manually mount the disks as EXT4 and could access the data so I figure this is a quirk of 16.04 I need to figure out.  So far so good!  I commented the two drives out of fstab and attempted a reboot, I got a bit further but ended up in maintenance mode again.

This time around I did some more digging and found the “apt –configure –a” command which reconfigures all the packages installed, this was recommended for interrupted installations and for me it worked a treat.  I could now boot normally!

As previously mentioned I use Greyhole for file duplication across my disks, for long-time readers of my blog or those familiar with it it’s very similar in concept to Drive Extender on Windows Home Server, Greyhole wasn’t happy.  First off it complained about PHP and MySQL errors so one by one searched for the error line and installed the missing packages.  After that I managed to get Greyhole running against the manually mounted disks and I’m now moving data around so I can reformat the two odd ones out that are listed as zfs_members so I can get them in line with the others.  That’s in progress and I’ll cover it in another post as this one has rambled on long enough.

It has certain been a learning experience and I’ve got nerd points from my colleagues for actually managing to fix a borked upgrade, apparently most people would just reinstall but I figured I’d have a stab at it.  For a certified Windows fanboy I’ve certainly come a long way!

Media Center Video Stutter and Windows Home Server “DEMigrator.exe” Service

I’ve been having an issue for a long time now where video hosted on WHS is played back in Media Center and seemingly randomly the video stutters or stops completely and the UI becomes unresponsive for a while.  After a lot of digging and troubleshooting both in Media Center and Windows Home Server I discovered a simple trick which laid the issue bare.

When the issue occurs you can look at task manager on WHS then it may appear that there is little CPU utilisation, the trick here is to add the columns “IO Read”, “IO Write” and “IO Other”.  You will likely see that DEMigrator.exe has numbers increasing in “IO Other” when the stuttering is happening.  I implemented a fix from the We Got Served Forums and it seems to have worked a treat!  I followed the workaround on the page, repeated here for my records and as it will be easier for me to find and pass on. 🙂

The fix is very simple to implement, just create two text files with the contents below and rename them as indicated.  Create two scheduled tasks, one for on and one for off, and set them to run over night.  I have mine set so DEMigrator is running for three hours between 3am-6am.  You will get an error about the service not running but other than that all should be well.

DemigratorOff.bat

@echo off
sc stop "DriveExtenderMigrator"
sc config "DriveExtenderMigrator" start= disabled
exit

DemigratorOn.bat
@echo off
sc config "DriveExtenderMigrator" start= demand
sc start "DriveExtenderMigrator"
exit

Enjoy!

Keegan

Home Server Rebuild

My Home Server has served me well however the aging 1.2GHz VIA board that was in it has long since peaked.  Time for an upgrade…

I decided to go for the POV MB330-1 which has 4xSATA on board along with a eSATA and a dual core 1.6GHz Atom processor.  I also decided to add an internal 2.5” drive to act as the system disc to make hot swapping drives in my Chenbro case far easier.  I’m not going to bother documenting the build in too much detail, for the most part I simply put all the bit in a case so it shouldn’t be too taxing if you RTFM.  I installed the system using a USB key, details can be found on We Got Served.

IMAG0143 IMAG0142

Picture on the left shows the laptop hard disk hidden away in the lesser known internal slot in the Chenbro case, on the right we see the PCI SATA card to add the extra port for the system disc.  All four bays are on the mainboards SATA ports, I’ve no idea if this is best practice but it seemed to make sense that they be adjacent on the system to reduce a potential bottleneck at PCI.  It may also simply be my OCD…

One thing that had me baffled (and still does) is the extra SATA port situated near the eSATA port.  I tried plugging a disk into it and nothing happened, whatever it is I’m sorry to say it is not a SATA port.  Maybe it will be enable in a future bios update but for now its purpose remains unclear.

One piece of advice I cannot stress enough it to get the latest drivers from Realtek for the LAN.  I am not certain of the exact chipset used but these drivers worked for me.  Network throughput went through the roof after this and the connection became a lot more stable too.

Hope this helps someone, any questions feel free to ask in the comments.

Keegan

Ps.  No, I don’t work for Mini-ITX.com or get commission (a shame!) but I’ve been using them for years and they’ve been helpful with any questions I’ve had.

UPDATE:  If you randomly get bluescreens or messages saying "You system has recovered from a serious error" try uninstalling IPv6.  My server was crashing a few times a day and after a kind soul at work reviewed the memory dumps it was IPv6 making the network driver crap out.

Windows Home Server and Windows 7 Network Throughput

I’ve a gigabit network at home and my transfer rates sucked, considering I frequently watch HD content from my Home Server this was less than ideal.  Having a chat with a few of my colleagues at work and after performing a simple tweak on my Home Server, throughput went from 30Mbps peak (that’s mega bits not mega bytes before anyone gets shirty) to a fairly stable 80~90Mbps.  A hefty improvement considering it involved the creation of a registry key and a restart!

Essential it involves enabling Compound TCP and I followed the steps to create the DWORD on this page.  If your Home Server is up to date then you don’t need to install the hotfix, only create the registry entry.

Mstsc to your Home Server and open regedit, don’t forget to backup your registry blah blah blah, and find the following key;

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersTCPCongestionControl

If it isn’t present, create a DWORD of the name TCPCongestionControl and set it the value of 1, reboot and you should be good to go.

UPDATE:  This hasn’t been a permanent fix for me, if anyone has any other tips feel free to add them as a comment.
UPDATE UPDATE:  Inbox Realtek drivers suck ass, www.realtek.com.tw is your friend…

Enjoy,
Keegan