Customizing View Disks Tab

[joeschmuck]

I'm working again on making some changes to FreeNAS for my personal use and right now I'm trying to add a few columns of data to the Storage -> View Disks tab. The columns I am adding are: Temp, Run Hours, Spinup Cycles and Pending Sector Count.

The work I've done to date:
I have made some minor changes to the layout, now the columns are:

Code:

|Edit | Name | Serial # | Description | Temp | Run Hours | Spinup Cycles | Pend. Sect. Ct. | HDD Stby | APM | AL | SMART On |

The code to derive the data is not complete but here is the idea:

Code:

smartctl -a /dev/ada1 | grep 'Temperature_Celsius' | sed 's/\(..\)*/\1/'
smartctl -a /dev/ada1 | grep 'Power_On_Hours' | sed 's/\(....\)*/\1/'
smartctl -a /dev/ada1 | grep 'Pending' | sed 's/\(....\)*/\1/'
smartctl -a /dev/ada1 | grep 'Start_Stop_Count' | sed 's/\(....\)*/\1/'

I believe I need to make changes only to middleware.py to add some version of the code above in order to pass the values to view.py but this is where I get lost quickly.

Once I figure this out I will post it but I'm looking for some guidance right now because after 2 days I'm not gaining ground fast.

--- Updated ---

I've also located 0029_auto_add_field_disk_serial.py where I added into storage.disk

Code:

'disk_temp': ('django.db.models.fields.CharField', [], {'max_length': '5'}),

since I'm working on Temp right now. Don't want to bite off too much.

 

[ProtoSD]

Mark, I don't have time to dig into the code to help you right now, but could I suggest something a little cleaner than using 2 commands (grep & sed) to get the values you're after? Here's an example:

Code:

smartctl -a /dev/ada1 | awk '/Temperature/ {print $10}'

EDIT:

I know you're focused on the other stuff at the moment, but this example below works even better since it only runs the smartctl command once and stores the results in a variable. Should speed up things a bit.

Code:

#!/bin/sh

RESULTS=$(smartctl -a /dev/ada1)

disk_temp=$(echo "${RESULTS}" | awk '/Temperature/ {print $10}')
power_hours=$(echo "${RESULTS}" | awk '/Power_On_Hours/ {print $10}')
pending=$(echo "${RESULTS}" | awk '/Pending/ {print $10}')
start_stop_cnt=$(echo "${RESULTS}" | awk '/Start_Stop_Count/ {print $10}')

echo ${disk_temp}
echo ${power_hours}
echo ${pending}
echo ${start_stop_cnt}

 

[joeschmuck]

Thanks for the advice. I was only going to run smartctl once and had coded that last night but I didn't paste that portion at the time and I appreciate your fine example since awk is so much cleaner. I was looking at awk this morning but lets be honest, I don't know this language and I've never claimed to. Google is my friend here and so are the folks in this forum.

 

[paleoN]

I've also located 0029_auto_add_field_disk_serial.py where I added into storage.disk

Code:

'disk_temp': ('django.db.models.fields.CharField', [], {'max_length': '5'}),

since I'm working on Temp right now. Don't want to bite off too much.

Why are you adding a db field for this? I assume you want the page to query the HD temps whenever you view it.

If you want to graph the temps in the future a single db field isn't going to be helpful anyway. You will want to use a rrd database ala the Reporting Tab.

 

[joeschmuck]

I thought the entry needed to be made as part of the storage.disk definition, because isn't it a database of drive data since there are typically multiple drives. I'm not looking to graph the temps, I only want to look at the current temps when I open the view disks tab, however a graph would be a nice touch but it may be too much to bite off for me right now. Honestly once I'm able to put a temp on the screen, I'll be happy, but if I understand how I got there, happier.

 

[paleoN]

I thought the entry needed to be made as part of the storage.disk definition, because isn't it a database of drive data since there are typically multiple drives.

I'm not at all familiar with this section of the code. It's certainly not required, but it may be less work? You might be able to do something similar as def get_serial in storage/models.py for temps. In storage/views.py - def disks_datagrid appears to pass the display name of the columns and the name of the corresponding db fields. Which is displayed via storage/datagrid_disks.html tempate. Not sure if disks_datagrid_json is only used for multipath devices or what.

Honestly once I'm able to put a temp on the screen, I'll be happy, but if I understand how I got there, happier.

I hear that. ;)

 

[joeschmuck]

This is a royal pain in the butt!

So I've been tracing how we pull the serial number off a hard drive and thinking I could use something similar to pull off other SMART values, boy was I wrong! What a mess this code is and it's twisted all over the place. I would have shot myself if I programmed that way back in the late 70's. But then again programming was a bit different as well. I will also have to say that my lack of Django hurts me.

Here is what I've done to date (All changes are to 8.3.1 source code and I'm actually making changes directly on my NAS):

1) I have altered the gui/storage/views.py at line 262

Code:

def disks_datagrid(request):

    names = [
        _('Name'),
        _('Serial #'),
        _('Description'),
        _("Temp"),
        _("Run Hours"),
        _("PendSectErr"),
        _("HDD Standby"),
        _("  APM"),
        _("   AL"),
        _("SMART On")]
    _n = [
        'devname',
        'disk_serial',
        'disk_description',
        'disk_temperature',
        'disk_runhours',
        'disk_pendsecterr',
        'disk_hddstandby',
        'disk_advpowermgmt',
        'disk_acousticlevel',
        'disk_togglesmart',
        ]

2) I started searching for all code related to obtaining the serial number of a device, what a mess! I settled on a few things, disk_serial, get_serial and serial_from_device.

3) "disk_serial" was my starting place because it's listed directing in the views.py so I know I need to make alterations there but where is the question since it resides in 44 of the files. Most of these are in "gui\storage\migrations\" and I don't think those are the ones I need to look at, but the others are listed below and I feel I need to do something in the notifier.py for certain and maybe the 'class Disk(Model):' in models.py:
..gui\middleware\notifier.py
..gui\storage\forms.py
..gui\storage\models.py
..gui\storage\views.py

4) "get_serial" is in gui\storage\models.py and I added the following at line 464:

Code:

def get_temperature(self):
        n = notifier()
        return n.temperature_from_device(
            n.identifier_to_device(self.disk_identifier)
            )

5) "serial_from_device" is in multiple locations:
..gui\middleware\notifier.py
..gui\services\migrations\0029_iscsi_name_changing.py
..gui\services\migrations\0035_device_extents_changing.py
..gui\services\migrations\0036_identify_dextents_serial_or_devname.py
..gui\services\migrations\0043_iscsi_extents.py
..gui\storage\models.py
..gui\storage\migrations\0008_auto__add_field_disk_disk_identifier.py
..gui\storage\migrations\0012_name_changing.py

6) I added to gui\services\migrations\0036_identify_dextents_serial_or_devname.py at line 17, and yes I know I'm still using serial for the value and that is fine, right now I need to get a value, any value on the View Disks tab.

Code:

def temperature_from_device(self, devname):
    p1 = Popen(["/usr/local/sbin/smartctl", "-i", "/dev/%s" % devname], stdout=PIPE)
    output = p1.communicate()[0]
    search = re.search(r'^Serial Number:[ \t\s]+(?P<serial>.+)', output, re.I|re.M)
    if search:
        return search.group("temperature")
    return None

7) I tried adding code to gui\middleware\notifier.py and duplicating the serial_from_device but changing the first line to the following but nothing good happened, it would generate errors in several modules when restarting Django.

Code:

def temperature_from_device(self, devname):

So this is where I am, confused. I'll end up working on this some more tomorrow but right now it's time to turn off the computer. I still have a mailbox I need to rebuild. Jerks came by and beat up the mailboxes in my neighborhood. Mine survived but many didn't.

Cheers,
Mark

 

[cyberjock]

Maybe I'm completely out to lunch here. I have no idea how much of an undertaking this is(and its probably going to show before the end of this paragraph) but why not just have a separate page with all of your attributes? Inject an extra button on the View Disks page that redirects to your new page and then enjoy your new blank page to do with as you please?

 

[joeschmuck]

The reason I chose the View Disks tab was because it already shows all the hard drives and it was easy to add a few new columns and steal from the serial number code. I would think that would be easier and I wouldn't have to recreate too much stuff. And adding a new button to open a new tab might be easy for someone who knows how to do it here, I'm at a loss so I took the best option I though I had.

I wanted to work on a different approach today but after reflecting on it, It just might screw up my pool. I was thinking about adjusting the serial number section until I could enter a static value. Hum, I might be able to do that with a VM. The problem with a VM is I don't get serial number data for the VM drives, maybe I could induce one. If I find that section of the code, maybe I'll figure it out, but right now I'm not very hopeful. Like I said, the code is a mess compared to my younger days of programming, but it works apparently fine so the end results are great.

It would be nice to have a breakdown of all the code and it's functions, I am trying to create a general list but there is a lot of code.

 

[paleoN]

What's the output of:

Code:

sqlite3 /data/freenas-v1.db '.schema storage_disk'

 

[joeschmuck]

What's the output of:

Code:

sqlite3 /data/freenas-v1.db '.schema storage_disk'

That gives me all the values from the 0029_auto_add_field_disk_serial.py file that I mentioned earlier. I really don't want to make it part of the freenas-v1.db file, do I? Won't this write to the bootable Flash drive when a value it updated? I would think the SMART values should be stored in a temporary file in /var.

EDIT: If I'm being a rock, I apologize.

 

[paleoN]

EDIT: If I'm being a rock, I apologize.

Not sure I get the reference. I was asking for the actual current output though.

I really don't want to make it part of the freenas-v1.db file, do I? Won't this write to the bootable Flash drive when a value it updated? I would think the SMART values should be stored in a temporary file in /var.

Yes, /data is the s4 partition and it is regularly written to. I'm all for a temporary file in /var, but that appears to be a bit more work from when I glanced at the code.

Have you worked your way through the Django tuturial? Now, you don't have to, but I'd argue it's the less painful approach.

 

[joeschmuck]

Sorry, here is the data from the command. I thought you were trying to point me in a direction. The rock comment was about me being too stupid to follow this type of programming. I tried locating a Django book in the book store, none to be found so I need to locate a good book on the topic, maybe "An Idiots Guide to Django".

Code:

[root@freenas] ~# sqlite3 /data/freenas-v1.db '.schema storage_disk'

CREATE TABLE "storage_disk" ("disk_enabled" bool NOT NULL DEFAULT 1, "disk_acousticlevel" varchar(120) NOT NULL DEFAULT 'Disabled', "disk_hddstandby" varchar(120) NOT NULL DEFAULT 'Always On', "disk_serial" varchar(30) NOT NULL DEFAULT '', "disk_multipath_name" varchar(30) NOT NULL DEFAULT '', "disk_identifier" varchar(42) NOT NULL DEFAULT '', "disk_togglesmart" bool NOT NULL DEFAULT 1, "disk_advpowermgmt" varchar(120) NOT NULL DEFAULT 'Disabled', "disk_transfermode" varchar(120) NOT NULL DEFAULT 'Auto', "disk_multipath_member" varchar(30) NOT NULL DEFAULT '', "disk_description" varchar(120) NOT NULL, "disk_smartoptions" varchar(120) NOT NULL, "id" integer PRIMARY KEY, "disk_name" varchar(120) NOT NULL DEFAULT '');

 

[paleoN]

Sorry, here is the data from the command. I thought you were trying to point me in a direction.

Both, but you didn't seem to be pointed the right way.

The rock comment was about me being too stupid to follow this type of programming. I tried locating a Django book in the book store, none to be found so I need to locate a good book on the topic, maybe "An Idiots Guide to Django".

I suppose rocks are dense. ;) I'd argue it's worthwhile to follow along, by which I mean actually run the commands, of the official Django tutorial. The other route is to smash your face into a concrete wall ... repeatedly ... over several days/weeks. I can tell you it's not enjoyable.

In storage/views.py - def disks_datagrid appears to pass the display name of the columns and the name of the corresponding db fields. Which is displayed via storage/datagrid_disks.html tempate. Not sure if disks_datagrid_json is only used for multipath devices or what.

Based on the output you haven't updated the db with the additional fields. From above, how I read storage/datagrid_disks.html, it's using Django to display the value of the db fields passed to it. You're currently telling it to use non-existent fields which doesn't work.

 

[joeschmuck]

Oh sorry, I reverted many of the changes I had made and restored things to original for the most part.

Thank you for the tutorial link, I will take your advice and go through this first but using a VM of course. Let's see how long this takes me to get through. Have you seen the movie The Croods? All I can say is I've taken a few snap shot photos already.

 

[paleoN]

Oh sorry, I reverted many of the changes I had made and restored things to original for the most part.

OK.

Have you seen the movie The Croods? All I can say is I've taken a few snap shot photos already.

No and I was unimpressed with the previews.

 

[MaIakai]

Please please add Disk Model number too!

 

[Rand]

Joe (Mark actually as i've just seen;)),

have you ever completed this nice little project?;)

Thats something i was thinking about as well and while i have no experience at all with Django nor Python thats not a reason not to try it ;)
If you're not finished and need help i meant:)