Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 310 311 [312] 313 314 ... 796

Author Topic: if self.isCoder(): post() #Programming Thread  (Read 907033 times)

Stargrasper

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4665 on: July 12, 2013, 06:30:12 pm »

Since I need a new excuse to procrastinate on my homework and to get myself into the practice of writing again, I thought I'd start with a long-promised brief lesson on makefiles.

Please be aware that you are expect to have a basic understanding of a *nix terminal for this lesson.

Stargrasper's Makefile Overview

Make is a Unix utility that was originally designed to streamline and automate the building process of applications.  We can also use it to automate an assortment of other tasks.  Commonly, these will be tasks such as cleaning up the workspace by deleting generated files or packaging the application for distribution.

In the simplest of circumstances, you can invoke Make by typing the following:
Code: [Select]
make

Make needs a set of rules to follow in order to understand what it is you are asking it to do.  The file that contains this information is known as a makefile.  When called with no other arguments, make will search for a file called GNUmakefile, makefile, or Makefile.  In that order.  Unless you tell it otherwise, if files by more than one of those names exists in the present working directory, make runs the first that it encounters.  So if makefile and Makefile are both in the present directory, make will run makefile, but not Makefile because it searches first for GNUmakefile, if not found, then makefile, if not found, then Makefile.

If, for whatever reason, you want make to run something other one of the default makefiles, you can do that.
Code: [Select]
make -f other_makefile

The -f argument tells make that you want to run a specific makefile; named whatever you want and located where you want.

Similar to a code file, a makefile can contains variables and commands.  But to know which commands to run when, the makefile has what are referred to as rules.  A rule consists of a rule name, a colon, and a space delimited list of prerequisites to run that rule.
Code: [Select]
rule_name : this is a list of prereqs

On the following line is the recipe, or command, that will be run when the rule is invoked.
Code: [Select]
application : main.c main.h
gcc -Wall -lm -o application main.c

Note that the command begins with a tab.  A lot of programmers use spaces instead of tabs while writing.  Make will not allow you to do this.  Thankfully, make will give a very clear error explaining what you've done wrong and how to fix it.  Also, be aware that you can run any command here that you can run in your terminal.  You only get one line, though.  You can run multiple commands.  There's a couple of ways.  You can have the rule run another rule through creative abuse of prerequisites.  You can also separate commands with a semicolon, just like you can in the terminal.
Code: [Select]
application : main.c main.h
cd src; make

And this is another trick you can use.  Running one makefile from another.  Just as you may split code into multiple files, you may split makefiles into multiple files by creatively running them all from a master makefile.  This has limited utility, but I've done it before.

You can also have rules call other rules.
Code: [Select]
application : main.o
gcc -Wall -lm -o application main.o

main.o : main.c main.h
gcc -c -Wall -lm -o main.o main.c

If multiple rules exist and you don't tell make otherwise, it runs the first rule it finds.  In this case, the application rule.  Then it looks at the dependencies and sees that it needs a file called main.o in order to run this rule.  Wait...what do you mean no file called main.o exists?  Well, I have a rule that creates such a file, so lets run that.  Simply put, if a rule calls for a file that doesn't exist, make looks for a rule that creates such a file.  If such a rule exists, that is invoked.  If such a rule does not exist, then make errors out.

You can also elect to run specific rules.  For example:
Code: [Select]
make main.o

This would invoke specifically the main.o rule.  I could also explicitly call the application rule, but because it is the default rule, this isn't necessary.

Now, lets look at this makefile.
Code: [Select]
application : util.o main.o
gcc -Wall -lm -o application util.o main.o

main.o : main.c main.h
gcc -c -Wall -o main.o main.c

util.o : util.c util.h
gcc -c -Wall -o main.o main.c

When make runs, it runs the default rule, application.  It sees that it needs util.o before it can run the rule, recognizes the file does not exist, and looks for a rule that creates the file.  Then it suspends running application to run util.o and checks that util.c and util.h exist.  It compiles the util.o object that returns to the application rule and sees that main.o doesn't exist.  So it suspends application again to run the main.o rule.  Then it finally is able to run the application rule.

Now, if you're familiar with gcc/g++, you've probably noticed my last couple examples have done all my linking in the default rule and splitting compiling in other rules.  Why would you want to do that?  Because make is smart.  If the file created by a rule exists when that rule is run, make checks the time stamp of the created file against the time stamps on the prerequisites.  If the created file's time stamp is newer, that is, none of the prerequisite files have changed since the created file was created, it will not run that rule.  So, if I compile the above with make and later change main.c, but nothing else, make will only recompile main.o and application.  Because the prerequisites for util.o have not changed since it was last compiled, it is unnecessary to recompile the file.  This means next to nothing in the baby programs you're probably writing.  It might mean something if your application has hundreds of source files, though.  This is one of the things that makes make really shine.

You can also touch a file.  Touch is a *nix command that creates a file if it does not exist or updates the timestamp if it does.  This has a handful of uses, depending what you want to touch.  It's a bit like updating a file without actually changing anything.

Now, we can create a rule that isn't compiling code.  Lets say we want to package our code.
Code: [Select]
app.tar : main.c main.h util.c util.h
tar -cvf app.tar main.c main.h util.c util.h

A tar file, sometimes called a tarball, or tape archive, places several files into one file originally intended for backup to a cassette tape.  Just as an aside, the reason you never see .tar files is that the .tar format does not compress data.  You frequently see either .tar.gz or .tar.bz or some other format appended to the end of files because they are compressed using some specific compression algorithm.

As you can see, we can create an archive file for distribution of our code.  Just like compiled files, if the prerequisites, anything in our code, have changed, the archive will be re-created when the rule is run.

Another creative technique we have is rules that perform an action that doesn't actually create anything.  Let's take a look.
Code: [Select]
.PHONY
clean :
\rm -rfv *.o application

A rule that does not create anything is called a phony rule.  Turns out, gnu make won't complain if you don't tell it a rule is a phony rule.  In this case, our clean rule deletes all of our object files and the application executable.  Why do this?  Before you distribute a new version of your program, traditionally, you whip out the object files and recompile it from scratch.  Why?  Sometimes weird things happen in files.  Something changes somewhere that affects a file in a manner you did not expect.  Recompiling completely can either eliminate or create the problem.  If the change is coming from a file you didn't expect, it likely isn't listed as a prerequisite for the first file and thus the first file isn't recompiled when the second is altered.  Most of the ways of avoiding this situation entirely are fairly advanced techniques.  For example, it's possible to ask gcc to work out and tell you all of the dependencies of a given code file.  We'll worry about doing that some other time.

For now, lets finally get to variables.  There are three types of variables that make will acknowledge.  User-defined variables types are recursively expanded variables and simply expanded variables and what are referred to as automatic variables.  Recursively expanded variables are expanded any time the variable is referenced using the variables that are referenced.  Every time that a recursively expanded variable is referenced, the variable is expanded again to find the current value of the variable at the time it is called.  Simply expanded variables are expanded once, meaning that they cannot actually contain any references to other variables.  At the time of definition, variables are expanded and the simply expanded variable contains the value of the variable referenced at the time it was referenced.  When I say "expanded", I mean, if the variable references another variable, it will resolve the value of that variable.  Recursively expanded variables are set with an '=' sign, while simply expanded variables are set with a ":=".

Automatic variables are set for you by make.  Most of these are extremely cryptic and I'm only going to cover a small number of them.
Code: [Select]
CC := gcc
CFLAGS := -c -Wall
LFLAGS := -Wall -lm
APP := application
SRC := *.c
SRCHDR = $(SRC):.c:.h)
OBJ := $(addprefix .obj/, $(SRC:.c:.o))

all : $(OBJ)
$(CC) $(LFLAGS) -o $(APP) $^

.obj/%.o : %.c %.h
$(CC) $(CFLAGS) -o $@ $^

app.tar : $(SRC) $(SRCHDR)
tar -cvf $@ $^

.PHONY
clean :
\rm -rfv *.o $(APP)

If THIS doesn't look cryptic to you, then you probably don't need to be reading this.  Lets just walk through this...

At the top are my variable declarations.  You'll notice I do a lot of simply expanded variables.  No particular reason than habit in this case.  SRCHDR and OBJ are my only variables that need to be expanded and because I never change SRC, which is what they depend on, the expansion never needs to change.

CC is a traditional variable for your C compiler.  The equivalent traditional C++ variable is CXX.  CFLAGS are your compile flags, while LFLAGS are your linker flags.  Remember that compilation is three steps; compile, link, assemble.  We're splitting them up a little here.  The equivalent C++ tradition is CXXFLAGS and LXXFLAGS.  These are fairly common variables and it makes life easier to make them variables for ease of access and alteration.  For instance, if took this makefile and tried to run it on Windows with cygwin or mingw, it'll spit an error in your face.  Well, with a variable, it's easy to fix.  Change gcc to cygwin/mingw/insert-compiler-here.  And it changes it everywhere.  Your compiler and linker flags can be just as easily modified.  Just don't remove the -c from your compiler flags unless you want an error during linking.

The APP variable is simply the name of the executable that will be compiled.  Now really, for what reason would the name of the final binary need to be a variable?  It's, in all likelihood, getting called twice.  During final linking and assembly...and during cleaning.  Well, there's an old saying: There is only one makefile.  Every other makefile ever to exist is a copy derived from the first makefile.  Nobody every writes these things from scratch.  Having such a variable will help re-usability when you want to copy the makefile to another project with a minimum of things that have to be changed.  Ah, the SRC variable.  The list of all source files.  If all of your source files are in the same directory as your makefile, this variable will scan them all for you.  Alternatively, if you want to put them somewhere else and don't want to fix the next two variables afterward, run this command in your terminal.
Code: [Select]
ls ./src/ | grep .c

That'll give you a list of all your .c files that you can copy and paste onto that line.  Not the most direct way of doing this, but probably the easiest.

The next variable, SRCHDR, is a little hack that takes everything in the SRC list and pastes it into the SRCHDR list, but replacing every .c with a .h.  This does assume that every .c file has a corresponding .h file and that no other .h files exist.  It can be modified accordingly if this isn't true.  For no reason, SRCHDR is also the only recursively expanded variable I have.  As stated earlier, in this example, nothing really needs to be recursively expanded and I'm trying to leave you with a makefile you can use, so I'm not adding too many theoretical examples.  Last variable is OBJ and boy is this magic.  We grab every element of the SRC list, pre-pend .obj/ to it, and then replace the .c with a .o.  This is an extremely useful technique that I strongly advise using because it allows a shortcut a little bit later.

You probably also noticed that you can't just use the name of a variable to directly call it.  To call a makefile variable, you use $(NAME) where NAME is the name of the variable in question.  As you can see, you also have limited ability to modify a variable within the call to it.

For the default rule, I usually just call it all because I usually use the default rule to say "build everything".  And since it depends on every .o, every .o needs to be compiled or recompiled.  The recipe line is a bunch of variable calls.  Back to that in a moment.

The second rule is where black magic is being used everywhere.  First, make allows different types of wildcards.  You saw above the * means "everything".  Or in the case we used it, "every that contains .c".  Technically, I guess we should have used '^*.c&'.  It's a regular expression thing.  I'll write a regex lesson some other time.  In reality, any wildcard that works in a bash shell, should work in a makefile.  Another type of wildcard make allows is %.  This is a special wildcard in that it can only be called on one thing at a time.  In our case, the all rule will fail when it finds a .o file that doesn't exist.  It sees a rule that can create one and replaces the % with the name of one of the .o files it needs.  Everywhere in that particular run of the rule, all of the % signs are replaced with the same thing.  And then come the next .o file, they're replaced with something else.  So this rule gets called for every .o file needed, and when it is, every % is replaced the same way.  So when we need main.o, every % is replaced with main, giving us main.o, main.c, and main.h.

On the next line, we call the compiler and compiler flags, but what are these weird signs at the end?  They're automatic variables, it turns out.  $@ is an extremely handy automatic variable that is equivalent to the name of the rule it is called in.  I usually name rules to be the file they output and then use $@ as the output name.  $^ is another handy rule that means all of the dependencies.  $+ is the same thing, but if dependencies are listed multiple times, these duplicates are preserved.  Sometimes useful for linking.  Whatever list you wrote on the dependencies above, that's what this is.  Similarly, $< is the first dependency listed.  Still similarly is $?, which is every dependency listed that is newer than the target.  I see limited use for this one.  There's more automatic variables, but these are the simpler ones.  $@ and $^ are the ones I tend to use all the time.

The default rule should make a bit more sense now. It says compile with the given linker flags a file called application using the following dependencies.  The tar rule should make as much sense as well.  Create a basic tar file called application.tar containing all the source and header files.  Clean should also pretty much make sense.  Remove all .o files and the executable.  The \ before rm means to run the base command, not your alias.  Mind, I usually alias rm to 'rm -rfv' anyway, but a lot of people will alias it, but in a different way.

Now that you understand the cryptic automatic variables and variable manipulation, this probably makes a lot more sense now.  And, while simple, this is very much a complete makefile.  I've used similar makefiles many a time.  In fact, this is exactly a makefile I've used a couple times except that I changed the SRC variable to be *.c instead of listing them explicitly, which probably is a good idea anyway.  In fact, do that instead.

Now, if you've ever used an IDE for a compiled language, like C/C++ or Java, you've noticed that they don't make you write makefiles or other build scripts.  They automatically generate them.  One of the ways this can be done is with autotools.  These include autoconf and automake.  For a bit more cross-platform solution, you could use cmake, which is capable of generating build scripts for many platforms, including VisualStudio and ordinary Unix makefiles.  For Java, there's ant and maven.  These sorts of tools become borderline necessary once your project balloons to a few dozen individual source files.  An explanation is probably beyond the scope of this overview.

Please be aware that while these are based on previous makefiles I've written, I didn't have enough spare code laying about to actually test everything.  Everything should work.  If for some reason you find something to not function as you would expect, please let me know.  If people want, I may expand on this at some point as it turns out it really is an overview that doesn't go that deeply into writing makefiles.  I encourage questions and comments and some form of acknowledgement that somebody besides me read this.
Logged

P(ony)SI

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4666 on: July 12, 2013, 07:25:07 pm »

So, I'm making an RPG in Java using LWJGL and Slick2D.
I'm programming like a boss until I start giving colors to enemies so that there can be more variety.
Despite me assigning different colors to my testing enemies, they all turn out white.
I do some debugging, and I figure out it's because the color variable isn't being initialized.
I go on a wild goose chase, trying to find where the problem is. I look in the saving system, I look at the classes responsible for holding the enemies, but they're fine.
Then, I look in the enemy class, and I see that the color isn't initialized in the constructor. I add some code for that, and suddenly, the colors work.
herpderp

Also, I should totally make a Java tutorial sometime.
Logged

Telgin

  • Bay Watcher
  • Professional Programmer
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4667 on: July 13, 2013, 08:31:37 pm »

...makefile tutorial...

That looks like a good overview of makefiles and reminds me again why I'm glad to use IDEs with compiler integration anytime I can.  From what I understand many large software development studios that deal with compiled languages have at least one person dedicated to makefile writing and maintenance.  I've seen some really nasty ones in my time, but fortunately haven't ever had to create or modify such a thing.

In the spirit of tutorials, I'm slightly tempted to try writing one that would help newcomers integrate V8 into C++ like I did last week.  The documentation I've found online was spread all over the place, was inconsistent and very confusing at the best of times.  If I knew I was doing things correctly I'd want to help alleviate that lack of knowledge on the internet, but I'm pretty sure I'm doing things redundantly and circuitiously in a few places.  And I haven't yet tried to implement the C++ side of object lifetime management when the JS garbage collector collects C++ linked objects.
Logged
Through pain, I find wisdom.

Stargrasper

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4668 on: July 13, 2013, 10:20:43 pm »

...

Also, I should totally make a Java tutorial sometime.

I'll write more in the near-ish future...  I wanted to write something else before writing more Java.  If you also want to write Java lessons, then absolutely go for it.  I very strongly encourage you to do it.  But consider reading mine first and think how to differentiate yours.  Or don't read mine so you don't have any preconceived notion as to what you should do.  Either way would be productive.  Tutorials that say the same thing as has been said have limited usefulness, but a different point of view or focus can mean a lot.  For instance, a major focus for my tutorials has been conceptual detail.  But more importantly, don't just talk about doing it.  If you're interested in doing so, then actually do it.

...makefile tutorial...

That looks like a good overview of makefiles and reminds me again why I'm glad to use IDEs with compiler integration anytime I can.  From what I understand many large software development studios that deal with compiled languages have at least one person dedicated to makefile writing and maintenance.  I've seen some really nasty ones in my time, but fortunately haven't ever had to create or modify such a thing.

In the spirit of tutorials, I'm slightly tempted to try writing one that would help newcomers integrate V8 into C++ like I did last week.  The documentation I've found online was spread all over the place, was inconsistent and very confusing at the best of times.  If I knew I was doing things correctly I'd want to help alleviate that lack of knowledge on the internet, but I'm pretty sure I'm doing things redundantly and circuitiously in a few places.  And I haven't yet tried to implement the C++ side of object lifetime management when the JS garbage collector collects C++ linked objects.

Thanks for the compliment.  I just hope somebody here can actually use it.

If you want an exercise in makefiles, look at the makefile your IDE generates for you.  Being completely generated by the software, it's remarkably nightmarish.  I'm a little impressed they work.

If you think you can put knowledge into writing, try it.  Doing redundant or otherwise unusual things can be okay if it works.  If there's flaws, we'll likely see them and help correct them.

</kindof_pushy_encouragement>
Logged

alway

  • Bay Watcher
  • 🏳️‍⚧️
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4669 on: July 14, 2013, 12:28:41 pm »

This is a programming and coding thread though which doesn't really have anything to do with security hackers, so... why ask here?
I can't help but somewhat disagree with this. As programmers, we pretty much are the first and only line of defense against such things, and so it is incredibly important to at least know all the common and obvious attacks.

Specific attacks will vary by language and such, as they are themselves the result of an 'exploit' or 'bug' which allows more access in some way than was intended. As for a couple basic types of attacks which are probably the most common, simply because not enough programmers protect themselves against them, are these two:

1. Never allow the user to directly give you code to execute. It seems obvious, but this is probably the single most common way to get your shit hacked. Most of Anon and LulzSec's hacking was done through either this or social engineering. Even random internet script-kiddies can and will exploit this if you allow them to. Usually, this takes the form of SQL commands send into a server database; you have your website/app frontend which sends the correct commands, but simply by altering the information being sent, it's trivial to tell the database to do whatever you want it to. To avoid this, ALWAYS ALWAYS ALWAYS use a practice called 'sanitizing your data.' Instead of directly executing anything sent by the end user, you should always parse through the sent data and translate it into only those commands you want to allow your user to do. Otherwise you will end up with some clown coming through, stealing every piece of data in your databases, and then finishing up by deleting your entire database's contents.

2. Buffer overflows. Mostly in languages than can involve direct memory manipulation, there may occasionally be out of date functions which don't take buffer overflows into account. This is when you've got x bytes assigned to hold a bunch of data, but the function assigning data from one container to another doesn't check the bounds. The most well known example of this is C/C++'s strcpy() function. C-style strings are character arrays which are null character terminated, rather than specifically tracking their length. The problem with strcpy is it would got through the input string, one character/byte at a time, copying from the input to the output string. However, if the input string was longer than the output string... it would continue copying as normal. Except now it was overwriting other variables in the program's memory. Which allows whoever is giving that input string full access to modify data in your program. For example, in this article, the '(s)elf-exploitation' section: http://www.gamasutra.com/view/feature/194772/dirty_game_development_tricks.php
In which a game company literally used a buffer overflow in their EULA to download and install patches. Which tells you just how much power it gives you: you can literally install and run anything you want; like a virus for example. Which is why these are very dangerous if you have them accessible in your software. These days, most of these have been recognized and try to keep programmers from using them (IIRC, visual studio throws warnings when using the old version of strcpy() which tell you not to use them because of buffer overflows) and so they are less common. However, older libraries may still use them, so one should still be on their guard.

So those are the two really common ones which all programmers should be aware of and avoid at all costs (not that there are any costs associated with any of them).

As for the rest, the most powerful tools are zero-day exploits. These are software bugs which allow for some form of access; and are pretty much knowledge of 'how can I exploit this specific program to get it to do something for my virus/spyware/ect.' When companies find these, they get patched as soon as possible; but any sufficiently large code base will often have some. Additionally, companies are usually willing to pay on the order of $50k-$100k for revealing a new zero-day to them. But unfortunately...

Black hats pay more. We aren't talking your stereotypical basement-dwelling black-hat here; today, it's all about the organized crime and government agencies. The NSA and similar government organizations along with organized criminals both buy knowledge of these secret exploits, usually for prices over $100k each. They are then used to create viruses which target certain (usually very common) applications, and are pretty much what constitutes an effective virus. Stuxnet, for example, was revealed as a joint creation by the US and Israeli governments in order to disrupt the Iranian nuclear program by spinning up and destroying their centrifuges; and is credited with putting it back by over a year. It used something like half a dozen zero-day exploits, as well as being part of a wider virus espionage effort which lasted months in order to figure out how to get it inside. Flame, another recent virus, was another such joint effort between the US and Israelis, and is generally thought to be for the same end goals; though primarily for espionage rather than sabotage. It contained many zero-days as well, including 2 used by Stuxnet. Then there's Duqu; again targeted to Iran, and with very similar code to Stuxnet, but with an espionage purpose; authors currently unknown, but some good guesses can be made.

There's also cryptography stuff, like hashing & salting databases, but I recommend just taking a coursera course on Crytography to learn about those.

So that's the world we live in: One in which your bad code can open up your servers for attack, theft, and domination, and in which the bugs in your code are sold on an international black market for hundreds of thousands of dollars in order for governments and organized crime to exploit your programs for malware.
Logged

kaijyuu

  • Bay Watcher
  • Hrm...
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4670 on: July 14, 2013, 04:13:54 pm »

If we're making a fun little breakout clone in our spare time, I dunno if security is a big issue.


Certainly a big issue for like, database programming or something, though.
Logged
Quote from: Chesterton
For, in order that men should resist injustice, something more is necessary than that they should think injustice unpleasant. They must think injustice absurd; above all, they must think it startling. They must retain the violence of a virgin astonishment. When the pessimist looks at any infamy, it is to him, after all, only a repetition of the infamy of existence. But the optimist sees injustice as something discordant and unexpected, and it stings him into action.

alway

  • Bay Watcher
  • 🏳️‍⚧️
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4671 on: July 14, 2013, 10:41:07 pm »

Well, yes. Until you re-use some of your code in a later project because you know it worked fine and did what you wanted it to. Though it really only becomes an issue for networked stuff or stuff being deployed to a wide audience.
Logged

Siquo

  • Bay Watcher
  • Procedurally generated
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4672 on: July 15, 2013, 03:14:53 am »

This is a programming and coding thread though which doesn't really have anything to do with security hackers, so... why ask here?
I can't help but somewhat disagree with this. As programmers, we pretty much are the first and only line of defense against such things, and so it is incredibly important to at least know all the common and obvious attacks.

[snip sensible stuff]
All true, and we should know about all this, but working as a programmer close to people in networking and security, I noticed there's a large difference between us. They know some programming and lots of security (which is mostly about knowing where to get the latest news. This has led me to believe that being a security expert is more about who you know than what you know [1]), and I know lots a bit more of programming and a lot less of security.

So if you really want to get into security stuff, a programmer thread is still the wrong location to start. The only tip I can give is to be really careful, a lot of hackerwannabescriptkiddies get hacked or ripped off themselves in some way. Start with learning the basics, for when you're going into the underbelly (no, not 4chan) of the internet, you should be prepared.


Edit:
[1] Oh, and reading whitepapers. Endless amounts of whitepapers...
« Last Edit: July 15, 2013, 04:16:20 am by Siquo »
Logged

This one thread is mine. MIIIIINE!!! And it will remain a happy, friendly, encouraging place, whether you lot like it or not. 
will rena,eme sique to sique sxds-- siquo if sucessufil
(cant spel siqou a. every speling looks wroing (hate this))

MorleyDev

  • Bay Watcher
  • "It is not enough for it to just work."
    • View Profile
    • MorleyDev
Re: if self.isCoder(): post() #Programming Thread
« Reply #4673 on: July 15, 2013, 03:16:29 pm »

The basics are pretty easy to do. In college (UK 16-18 College), my friend has a php-based comments section he wrote himself on his website. He didn't sanitise the inputs, so I played a little joke and injected some javascript to redirect the page to a Rick-Roll. I told him about it more or less as soon as I did it. I basically did it to see if it would work, which didn't expect it to, but I guess he didn't expect anyone to bother trying. If I was a cruel person I could of easily injecting something more malicious or used some SQL injection and got ahold of his database.

As for security for programmers, the basics are don't be stupid. Sanitise inputs and use wrapping classes instead of dealing with data directly whenever you can that take care of concerns like buffer overflow for you automatically. People are fallible, so try and work in a way geared towards catching those failures that you will inevitably miss yourself. When not, be aware of what can happen.
« Last Edit: July 15, 2013, 03:19:55 pm by MorleyDev »
Logged

Siquo

  • Bay Watcher
  • Procedurally generated
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4674 on: July 17, 2013, 08:04:14 am »

Refactoring Database.
Hello table finance_mutation, let's take a look at your columns... Ah, there's id, mid, did, gid, kid, bid and rid. Because bytes are expensive!
Logged

This one thread is mine. MIIIIINE!!! And it will remain a happy, friendly, encouraging place, whether you lot like it or not. 
will rena,eme sique to sique sxds-- siquo if sucessufil
(cant spel siqou a. every speling looks wroing (hate this))

Telgin

  • Bay Watcher
  • Professional Programmer
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4675 on: July 17, 2013, 08:18:21 am »

Needlessly abbreviated identifiers are one of my pet peeves.  Sometimes I go to unnecessary extremes in avoiding it, but I really hate short, cryptic and convoluted identifiers.  If I never have to touch another FORTRAN program in my life...

Poorly formatted programs annoy me a lot in general and on our dev team I think I'm the one who cares the most about code formatting.  One other guy is at least consistent with his style, but his style is a little different from mine and that causes hiccups for me on occasion.  I'm OCD about my indentation style.

On the topic of badly formatted databases though, that is one thing we've got going for us here.  We are extremely consistent with our database structures and follow rules that make it a pleasure to write interfacing code.  I've had some moments of rage after attempting to maintain / interface with databases we didn't directly create.  My personal favorite was a table for a calendar system that went something like this:

id - Primary key
uid - User ID
note - Stored either who was assigned on a day, or unrelated notes (no, uid had nothing to do with who was assigned)
adminLEVEL - Naturally, this was a column of integers with no related table telling me what the numbers meant
add_date - Not a date or datetime column, just text

Consistency people... if nothing else, consistency...
Logged
Through pain, I find wisdom.

Mephisto

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4676 on: July 17, 2013, 08:57:17 am »

Our databases could probably be better. Currently, we just let Django manage them. We have a DBA, but I'm not sure what he does besides bring games for us to play on Fridays.


Anyway, I mentioned some time ago how bad most open source documentation is. Well, our own project's lack of documentation finally came back to bite us in the ass. This Monday, a pretty big part of our system went down and the two "experts" were out. Now I don't consider myself an expert, but I'm one of the two people who still work here and have touched that system before so I'm classified as an expert.

The person who originally set this up no longer works here and wrote no docs of any kind. I'm now taking part of the day to document and possibly place into source control this undocumented, untracked system. I actually like writing documentation, though, so it's okay.
Logged

MorleyDev

  • Bay Watcher
  • "It is not enough for it to just work."
    • View Profile
    • MorleyDev
Re: if self.isCoder(): post() #Programming Thread
« Reply #4677 on: July 17, 2013, 09:13:27 am »

We had a DBA once, but the developers write all the SQL now. There were some goliath stored procedures, but now the SQL has become just small commands and queries related to the updating and retrieving pieces of data, and the rest of the logic has a proper place not in stored procedures.
« Last Edit: July 17, 2013, 09:15:07 am by MorleyDev »
Logged

Stargrasper

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4678 on: July 18, 2013, 10:13:03 pm »

Stargrasper's Java Basics Tutorial Series Whatever-I-Was-Calling-It-Like-A-Year-And-A-Half-Ago

I posted the last lesson on January 22, 2012.  A year-and-a-half later, I'm posting this one.  In that time, I've learned a few things about being a programmer.  Let's see if all I've learned reflects in this lesson!

Now, I really wanted to just dive into objects.  Reading the old lessons, I got really tired of treating them as magic.  So, we're not going to do that anymore...okay, yes we will.  But not as much.  Okay, we actually have to talk about one more thing first.  You need to understand methods before you can understand objects.

Lesson 0.6.whatever: Methods

Methods, known as functions or procedures in most other programming languages, allow you to encapsulate code in a manageable space that can be inserted into almost any part of your code just by calling the method.  Here's a secret; that thing called main?  Yeah, that's a method.  A method that's required by Java desktop applications.  It tells them where to start executing code.  If you're writing a Java applet, the rules are different.  Thank you Sun Microsystems, now owned by Oracle, for a consistent environment.

So lets look at a tiny bit of code, shall we?
Code: [Select]
public static void main(String args[]) {
System.out.println("Hello World!");
}

The line public static void main(String args[]) is the method signature.  This means the application tells the compiler what to expect for input and output for that function; as well as other information on how the method can be called.  The public keyword reflects visibility of the method.  More on that in the objects lesson.  static and void are two more keywords that I'll get to shortly.  What we really want to talk about is main.  This is the name of the method.  What's inside the parenthesis tell the method what input to expect.  In this case, a String array.  This array is actually for command line argument.  If you're using an IDE, then getting at the option to pass command line arguments is pretty tricky most of the time.  If you're actually running from the command line, then it's trivial, naturally.  We talked about Strings and arrays previously, so you can review previous lessons or ask questions if need be.  Everything between the opening curly brace ('{') and the closing curly brace ('}') is the method body.  The method body contains the code that will be run when a function is called.

Let's write a function!
Code: [Select]
public static void hello() {
System.out.println("Hello World!");
}

public static void main(String args[]) {
hello();
}

We have two methods here.  The hello and main methods.  As stated earlier, code execution begins at the beginning of the main method.  Other methods do not get called unless you or an object call them.  How do you call them?  You say their name, of course!  To call a function, you write the name of the function you wish to call followed by parenthesis.  If the function has any input, it will go inside the parenthesis.  Like this:
Code: [Select]
public static void hello(String motd) {
System.out.println(motd);
}

public static void main(String args[]) {
hello("Hello World!");
}

In the main method, you pass the text string "Hello World!" by placing it within the parenthesis of the function call.  In my teaching experience, this next part is a little confusing to people.  Maybe I'm bad at explaining it, but it's one of those things few people I tutored caught quickly.  Kind of like booleans.

The hello method takes in a string variable called motd as a parameter.  When calling the method, the value of the first and only parameter becomes the value of the variable that is the first parameter of the method.  So the value of motd becomes "Hello World!" in this example.  It is equivalent to writing:
Code: [Select]
String motd = "Hello World!";

Now more magic I've been glossing over.  That System.out.println?  That's a method, too.  It takes in a String object.  In our example, we take the String motd and pass in to println as a method parameter.  println is a part of the Java standard library, so we don't have to define it ourselves.  We don't even have to include any extra libraries.

Remember that void keyword we skipped a few minutes ago?  Methods are designed to perform some set of operations for you.  If you want it to just do it and be done, like if we want it to print something, we use void.  But what if we want a function to do something and then give us a result?  We have to have a return value.  In the method signature, we replace void with the data type we want to return.
Code: [Select]
public static void a() {
// do something...
}
public static int b() {
int result = 0;
// do something...
return result;
}
public static String c() {
String result;
// do something...
return result;
}

Here are three functions.  a does not return anything.  b returns an int.  c returns a String.  In order to do this, we simply use the return keyword.  Now, it is very important to remember, return ends the function.  void functions end when they reach the end of the function.  You are allowed to have multiple return statements.
Code: [Select]
public static int translate(String aString) {
switch(aString) {
case "a": return 0;
case "b": return 1;
case "c": return 2;
default:  return -1;
}
}

Do you remember how switch statements work?  The switch statement looks at the variable you pass it as a parameter and searches it's cases for that value.  If not found, it uses the default case.  A minor aside; as of Java 7, you can switch on integers, enumerations, and strings.  Do you see how there are four return statements?  If a return ends the function, does this method end after the return 0?  No.  Because if you fail that case statement, you'll never actually hit that particular return statement.  But as soon as you hit any of the return statements, it's over.  return that value from the function.  You can use return statements in void functions as well.  It's a useful way to terminate the function early.  For example:
Code: [Select]
public static void foo() {
if(aCondition) return;
// do something...
}

Look closely.  We used return without actually returning anything.  This works because the function expects us to return void, which as far as Java is concerned, should be nothing.  Not Object.  Not null.  Not anything.  In fact, trying is a compile-time error.

Let's go a little crazy with methods.
Code: [Select]

public static int add(int n1, int n2) {
int result = n1 + n2;
System.out.println(result);
return result;
}

public static int subtract(int n1, int n2) {
int result = n1 - n2;
System.out.println(result);
return result;
}

public static int multiply(int n1, int n2) {
int result = n1 * n2;
System.out.println(result);
return result;
}

public static int divide(int n1, int n2) {
int result = n1 / n2;
System.out.println(result);
return result;
}

public static int equation() {
int result = divide(multiply(add(5, 2), subtract(7, 3)), multiply(8, 2));
return result;
}

public static void basicMath() {
// I want to add 2 + 2
// I wish I had a function to add
// Oh wait, I DO
add(2, 2);

// I want to subtract 5 - 2
// I wish I had a function to subtract
// Oh wait, I DO
subtract(5, 2);

// I want to multiply 8 * 4
// I wish I had a function to multiply
// Oh wait, I DO
multiply(8, 4);

// I want to divide 9 / 3
// I wish I had a function to divide
// Oh wait, I DO
divide(9, 3);
}

public static void multiCalls() {
// 4 + 8 + 2 + 8 + 1 + 6 + 4 + 5
System.out.println(add(add(add(4, 8), add(2, 8)), add(add(1, 6), add(4, 5))));

// 6 + 4 - 4 * 8 / 2 + 4 / 9 - 4
// Note the integer division
// kekekekeke
System.out.println(subtract(add(subtract(add(6, 4), divide(multiply(4, 8), 2)), divide(4, 9)), 4));
}

public static void run() {
// I want a function that demonstrates
// basic mathematical operations
// using functions
// I wish I had a function to do that
// Oh wait, I DO
basicMath();

// I want a function that demonstrates
// calling functions multiple times
// I wish I had a function to do that
// Oh wait, I DO
multiCalls();

// I want to know (5 + 2) * (7 - 3) / (8 * 2)
// I wish I had a function to do that
// Oh wait, I DO
System.out.println(equation());
}

public static void main(String args[]) {
// I want the program to run
// I wish I had a function to run
// Oh wait, I DO
run();
}

Okay, that was a touch more than a little crazy.  But it was FunTM!  Because some of this is complicated, lets walk through it as painfully slowly as we possibly can.

Now, you can clearly see that there are a lot of functions.  We wrote nine functions for this application!  In Java, it turns out that it really does not matter what order you place the functions in, unlike other languages like C/C++.  Does that make it better or worse?  Not really.  There's benefits and drawbacks.  As far as order goes, I like to place the main method at the very bottom.  This is for the sole purpose of making it easy to find.  Sure, I could just use a search, but I hate wasting time.  For stylistic purposes, there is an order I like to use.  It will make more sense to discuss that in the objects lesson, however.  Still, style is a matter of personal preference and professional requirements.  If your employer says "Do it this way", you darn well better.

Remember that your code begins execution at the main method.  Comments get ignored.  As far as the compiler is concerned, there's only one line here.  A function call.  And what does it do?  It moves us up into the run method.

The run method has four function calls.  But wait, there's only three lines of actual code in there!  Well, look closely at the third one.  Do you see it?  We embedded the equation method call inside of the println method call.  But we're getting ahead of ourselves.  The first function call steps us into the basicMath function...which contains four more function calls.  All of which take two ints, perform an operation on them, then print and return the resultant int.  Let's just briefly step into the add function.  We get to the end of it.  There are no more method calls for us to follow.  Now what do we do?  Now we step back out to where the function in question was called.  At the end of that function, we step out to where basicMath was called and advance to the next function.

In multiCalls, for lack of a better name, we demonstrate that you can keep calling the same methods over and over again.  The input parameters take the values of whatever was passed in at that particular method call.  Furthermore, this demonstrates embedding functions.  Look closely.  The add function returns an int.  Now we could just say int foo = add(2, 2).  But because it returns an int, we can essentially treat it as though it is an int.  This lets us pass it as a parameter to the next call to add.  For example, add(add(2, 2), 2), adds (2 + 2) + 2.  After seven calls to add, we print the result.

Let's walk through the method calls quickly.  The first thing the compiler sees is the println.  But it can't tell what we want it to print until it calculates all other method calls.  The compiler next sees the far left add, but it cannot tell what it is adding until calculating other method calls.  Next add, and the compiler still cannot tell what it is adding without calculating more calls.  Finally, at the third add, the compiler has a function it can complete.  At the end of that call, the call is, effectively, replaced with it's return value.  The same is true with the next add.  With those two methods complete, the compiler backs off to the second add and completes it.  Then the compiler has to do it all over again for the last three adds.  When they've all completed, the first add can complete and the end value can be printed.

Now, the next one is a bit complex.  We can't just group the numbers into twos anymore because of order of operations.  I'll be honest, this one took me a bit of thinking.  Mostly because I would never, ever, actually do it this way.

Walking through this nightmare...you know what, let's just do an old fashion trace.
Code: [Select]
run println(subtract(add(subtract(add(6, 4), divide(multiply(4, 8), 2)), divide(4, 9)), 4))
run subtract(add(subtract(add(6, 4), divide(multiply(4, 8), 2)), divide(4, 9)), 4)
run add(subtract(add(6, 4), divide(multiply(4, 8), 2)), divide(4, 9))
run subtract(add(6, 4), divide(multiply(4, 8), 2))
run add(6, 4)
return add == (10)
run divide(multiply(4, 8), 2)
run multiply(4, 8)
return multiply == (32)
return divide == (32 / 2 = 16)
return subtract == (10 - 16 = -6)
run divide(4, 9)
return divide == (4 / 9 = 0)
return add == (-6 + 0 = -6)
return subtract == (-6 - 4 = -10)
return println(-10)

Woo!  That was kind of hard.  But if you don't understand what's happening in the code, tracing out what's happening can be a good, if somewhat slow, tool.  You could also make it trace for you by putting print statements at the beginning of every function call, but that does not force you to walk through your code and understand it as intimately.  Especially if you're using an IDE, you might also consider using the built-in debugger.  Somewhere out there is a version of GDB that works for Java.  You could use that as well; if you're coding on the command line.  Step into every function you wrote and step over everything else.  And watch the value of variables while you are doing that.

Do you understand embedding methods yet?  If not, come up with some random mathematical formula that utilizes only the basic four operations and try to trace that out.  You should be able to make a single line equation with function calls.

Moving on, we're still in the middle of the program.  We step into the equation function.  This is another, albeit much simpler, example embedding function calls inside other function calls.  Let's just look at it quickly.
Code: [Select]
add(5, 2) = 7
subtract(7, 3) = 4
multiply(7, 4) = 28
multiply(8, 2) = 16
divide(28, 16) = 1

Okay, that was fast, but by now you should be able to understand how that method works.  Be sure to ask if you're not following!

After printing the value of that function, the program ends.  If you run the complete application, you should get the following output:
Code: [Select]
4
3
32
3
12
10
22
7
9
16
38
38
10
32
16
-6
0
-6
-10
-10
7
4
28
16
1
1

Not much to look at, but this leads to a good exercise.  Trace through the code and work out where each of those values came from.

We talked about scoping, right?  Anything declared within a set of curly braces stops existing outside of those braces.  For purposes of scoping, input parameters are local to body of the method they are a part of.  This is why we can keep using the variable names n1 and n2 for so many different unique variables.

We need a slight break, so let's talk technique for a minute.  Look at the code.  Do you see how nearly everything is a function call?  And how the functions that are not calling other functions are just a few lines long?  I wrote this using one of my favorite programming techniques for large projects.  Wishful thinking.  In short, every time you want your code to do something, you make up a function call.  "I wish I had a function that does THIS.  Oh wait, I DO."  Eventually, you get to the end of your program and have a skeleton worked out.  At the end, start filling in functions the same way.  "I need a function that does THIS.  Oh wait, HERE'S one!"  At the end of the day, your code is a large number of function calls that eventually lead to a large number of functions that have only a few lines of code.  Let's walk through the example.

I'm in the main.  Well, it'd be nice if the main would run the application.  Let's write a function that does that.  End of main, end of application.  Great!  Let's start filling in our functions.  What do I want run to do?  Well, I want it to give examples of using function calls for basic mathematics for the purpose of demonstrating the basic manner of writing and calling functions.  Then I want it to demonstrate the more advanced calling of the same function many times and embedding functions.  Then I want one more simple embedded function example for good measure.  Three things that I want run to do.  Gee, I wish I had functions that did that.  Wait a second, I DO!  Okay, end of run.  Step out to main.  End of that.  End of application.  Time to fill in functions again.  basicMath.  I want it to add, subtract, multiply, and divide with functions; both to demonstrate calling functions and to demonstrate writing them.  I wish I had functions that performed each of those operations.  Wait, I DO!  Step out.  Step into multiCalls.  I just want it to string together a bunch of adds and then show a complicated equation, using functions.  I could, and with wishful thinking, probably should, make that two functions.  I elect not to do that because I can actually do both with a single line.  And not a stupid single line like in an earlier lesson where I made the entire program one line.  This is a single line that, as an experienced programmer, I would write in any of my actual applications.  So I'll let this function break the rules of Wishful Thinking and just call the mathematical operator functions.  Step out.  Step into equation.  Not unlike the second example in multiCalls, this is a single line of mathematical operator functions strung together.  Assign it to a variable and return it.

I want you to do something.  It doesn't make sense right now, but don't question it for the moment.  Just do it.
Code: [Select]
public Methods() {

}

public static void main(String args[]) {
new Methods();
}

Why are we doing this?  To get us out of the main and the requirement that everything it touches be static.  What we've just done is instantiated a Methods object, but objects are still a bit magical.  We'll talk about them in the next lesson.  If I ever finish this one.  From here, just pretend that Methods is the main function for the time being.

This is actually something I do a lot when I'm coding Java applications.  Frequently, if I don't have a driver class, I use the main object to contain the main method.  Java requires that if you are in a static function, you can only call other static functions.  This makes perfect sense once you understand the difference between static and non-static functions.  Unfortunately, we're only going to touch on that.  It will make more sense after we talk about objects.

We're now going to write a number of static methods.  Wait, what?  Yes, I want to show you how you would normally use static functions outside the main method.
Code: [Select]
public static int add(int n1, int n2) {
return n1 + n2;
}

public static int subtract(int n1, int n2) {
return n1 - n2;
}

public static int multiply(int n1, int n2) {
return n1 * n2;
}

public static int divide(int n1, int n2) {
return n1 / n2;
}

public Methods() {
Methods.add(1, 2);
Methods.subtract(3, 4);
Methods.multiply(5, 6);
Methods.divide(7, 8);
}

public static void main(String args[]) {
new Methods();
}

The static functions look a touch different than last time, but they actually work the same except they no longer print the result in the function.  The purpose of static functions is be able to call functions that are not attached to a specific instance of an object.  For instance, there's really no reason that the above functions need to be associated with an object.  They can stand up completely on their own.  We'll talk more on the differences when we talk about objects, but I wanted to at least touch on this here.  In order to call a static function from a non-static function, you need to say the name of the class that contains the function, a dot, and then the function name as normal.

I can't believe I'm about to touch on this, but as long as we're here, lets briefly look at recursion, confuse everyone to all Hell, and then move on and talk about it more some other time.  You know what they say about recursion.

"In order to understand recursion, you must first understand recursion."

Code: [Select]
public static int fibonacci(int n) {
if (n == 0)
return 0;
else if (n == 1)
return 1;
else
return fibonacci(n - 1) + fibonacci(n - 2);
}

public static void main(String[] args) {
for (int i = 0; i < 100; ++i) {
System.out.print(fibonacci(i) + " ");
}
}

Take a look at fibonacci.  In the else statement, the method calls itself.  Yes, this method is, in fact, calling itself.  This is recursion.  Anything you can do with loops, you can do with recursion.  And vise versa.  Usually one will be much cleaner than the other.  There's a lot of problems with recursively generating fibonacci numbers, but we'll ignore those for now.  If you are going to use recursion, you need to create what are called base cases.  They are what halt the recursion, similar to a loop's continuation conditions.  Remember how you can have an infinite loop?  Well, you can infinitely recurse as well.  Except when you infinitely recurse, your program crashes with a StackOverflowErrorIf you want to see that happen, run the following code.  I'm not responsible for any damage this causes.  In my eclipse, this instantly crashes with a StackOverflowErrorCan't imagine why. </sarcasm>
Code: [Select]
public static int overflow(int n) {
return overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n)
+ overflow(n) + overflow(n) + overflow(n) + overflow(n);
}

public static void main(String[] args) {
overflow(Integer.MAX_VALUE);
}

You have to watch your base cases.  Usually these will be a few conditionals that stop you from recursing under certain circumstances.  Then just make sure that your recursion is always approaching your base cases.

It really is a good idea to split your code up into methods.  A single gigantic monolithic main tends to become unwieldy fairly quickly.  It also tends to lead to a very large about of duplicate code, which is typically a bad thing.  And there's other issues, but they will be easier to talk about after we discuss, you guessed it, objects.

* Somebody count how many times I wrote "function" when I was supposed to write "method".  Can tell I've been coding C/C++ and Python for the last year-and-a-half.

PS
I really wanted to talk about objects in this lesson.  But I just could not talk about objects without covering methods first.  And methods became a relatively lengthy lesson.  Then again, it's pretty much on par with my other lessons.

PPS
If anyone has any questions or comments, please feel free to express them.

PPPS
I think this might actually be one of my longest lessons.
Logged

Skyrunner

  • Bay Watcher
  • ?!?!
    • View Profile
    • Portfolio
Re: if self.isCoder(): post() #Programming Thread
« Reply #4679 on: July 18, 2013, 10:16:49 pm »

The order in which you write function declaration matter in C/C++? O_o That sounds like an oversimplification...
Logged

bay12 lower boards IRC:irc.darkmyst.org @ #bay12lb
"Oh, they never lie. They dissemble, evade, prevaricate, confoud, confuse, distract, obscure, subtly misrepresent and willfully misunderstand with what often appears to be a positively gleeful relish ... but they never lie" -- Look To Windward
Pages: 1 ... 310 311 [312] 313 314 ... 796