Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[tNok85]

Heya,

This is probably the best place for this... it's definitely a project, and arguably creative. The problem is as follows:

I have a fairly small sized set of data. There are approximately ~100 items to track, with twenty properties (numerical values) per item.

I want to fetch these items every 2 hours. I want to be able to compare properties between the items (highest to lowest, highest vs. lowest, etc) and also track trends - be able to pick say, the first property, and see the trend over the past 10 cycles (or more).

I can already export this data to an Excel spreadsheet. This seems like it might be more a database project, since I'd end up (eventually) tracking back many many sets of data - a week of this would be 84 sets of data, a month would be approximately 360 sets of data, and so on.

Anybody have any advice on which way to head with some sort of data management/tracking solution?

[Rysith]

This does sound like a database project. I'd probably start with MySQL or PostgreSQL, since those are both reasonably performant and free. SQL has built-in functions for all of the tracking that you mentioned, so implementing it doesn't seem that hard, and you're probably looking at being able to store hundreds to thousands of cycles at least before you have to start worrying about performance.

That's assuming that you're willing to write your own tracking software (though both of those, I believe, have user-friendly frontends in most major programming languages, or can be manipulated directly with SQL scripts if you're willing to learn a new language). You could also probably get someone to sell you such a product, but that would cost money rather than time.

[tNok85]

This does sound like a database project. I'd probably start with MySQL or PostgreSQL, since those are both reasonably performant and free. SQL has built-in functions for all of the tracking that you mentioned, so implementing it doesn't seem that hard, and you're probably looking at being able to store hundreds to thousands of cycles at least before you have to start worrying about performance.

That's assuming that you're willing to write your own tracking software (though both of those, I believe, have user-friendly frontends in most major programming languages, or can be manipulated directly with SQL scripts if you're willing to learn a new language). You could also probably get someone to sell you such a product, but that would cost money rather than time.

Great info - thank you. I don't mind learning a new language, one of the more fun things to do. I'll start looking into MySQL.


Edit: Would MySQL be able to read from an Excel document easily, or at worst a comma or space delimited text file?

[Nadaka]

Great info - thank you. I don't mind learning a new language, one of the more fun things to do. I'll start looking into MySQL.


Edit: Would MySQL be able to read from an Excel document easily, or at worst a comma or space delimited text file?

yes for CSV file, possibly XLS if you are very familiar with the file format.

http://dev.mysql.com/doc/refman/5.5/en/load-data.html

Edit: MySql will do just fine storing hundreds of thousands of records. PostgreSQL will do just fine storing millions of records. It is the size and complexity of your queries that will slow you down, with the amount of records you are talking about for a year, you should be fine for a couple centuries.

[tNok85]

Great info guys, thanks.

Any recommendations on starting books? I have reasonable experience with scripting/programming - LUA, Javascript, VB, Java, Python, some C++, and most recently been playing with AutoIt - great simple way to get most things done. Little to no experience with SQL, besides copying and pasting some queries...

(O'Reilly has always been pretty good to me)


Edit:
Also, I have a Windows box set up running Apache/PHP. Tossed MySQL/PHPMyAdmin on there now. PHPMyAdmin should be enough for a front end I hope. I wouldn't mind a nicer windows GUI (Navicat comes to mind, but that's a little pricey...), any suggestions or is PHPMyAdmin good enough?

[Rysith]

Great info guys, thanks.

Any recommendations on starting books? I have reasonable experience with scripting/programming - LUA, Javascript, VB, Java, Python, some C++, and most recently been playing with AutoIt - great simple way to get most things done. Little to no experience with SQL, besides copying and pasting some queries...

(O'Reilly has always been pretty good to me)


Edit:
Also, I have a Windows box set up running Apache/PHP. Tossed MySQL/PHPMyAdmin on there now. PHPMyAdmin should be enough for a front end I hope. I wouldn't mind a nicer windows GUI (Navicat comes to mind, but that's a little pricey...), any suggestions or is PHPMyAdmin good enough?

For the kinds of things you're talking about (trends for single items, highest/lowest for a single cycle), you could probably get away with just one of the SQL command references[1]. That's certainly how I've picked up what I know about hand-writing SQL. There's also a good amount of random advice floating around the internet.

[1]
http://dev.mysql.com/doc/refman/5.5/en/functions.html , for example