Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Sirus]

My only regret now is that my art skills aren't up to scratch for mocking up an Inquisitor of the Ordo Olīva.

+1

Actually, my only regret is ever tasting an olive, putrid things. Even worse than the vile flavour that is garlic.

You badmouth garlic now? May the Emperor have mercy on your soul.

[Bauglir]

I don't... I don't understand. Garlic... vile? I understand both of those words, but together they just don't seem to make any sense...

[scrdest]

My only regret now is that my art skills aren't up to scratch for mocking up an Inquisitor of the Ordo Olīva.

+1

Actually, my only regret is ever tasting an olive, putrid things. Even worse than the vile flavour that is garlic.

Giglamesh confirmed for vampire.

[GiglameshDespair]

I always wondered why I couldn't cross running water and regularly drank the blood of innocent maidens.

[LordSlowpoke]

and regularly drank the blood of innocent maidens.

could you fucking not i swear i haven't seen innocent maidens for dozens of years now you're driving them extinct

[LeoLeonardoIII]

There may be another reason you see fewer innocent maidens these days.

[Xantalos]

There may be another reason you see fewer innocent maidens these days.

That'd be the dragons.

[LeoLeonardoIII]

I ... yes. That is the thing I was thinking of.
<_<

>_>

[Telgin]

I screwed up pretty bad at work today.  An oversight caused some automated scripts I wrote on one of our servers to misbehave.  And by misbehave, I mean they didn't terminate properly and started spamming our customers with numerous and excessive duplicate text messages.  One poor soul got 38 copies of the message he/she was supposed to get, and nearly a thousand others got in the neighborhood of 20x as many copies as they were supposed to get.

The implications may not be obvious, but this could have had (and still may have) the potential to damage the reputation of our partner, who we were sending the text messages for.  That in turn could damage the relationship they have with us, and to put it simply, our company has become dependent on this partnership.

By no means do I expect this to cause our companies to part ways or even have serious repercussions, but it could have, and that's a little scary.  If I hadn't been watching the logs and caught it after 15 minutes, this could have been an absolute disaster.

At the very least, it did cost our company about $250 since the text messages aren't free.

Surprisingly, nobody got all that mad.  No threats to even dock my pay or anything.  I tremble at the thought of this having run all night.  It could have cost the company many thousands of dollars at least and possibly caused permanent damage to our partnership.  In fact, I'm kind of afraid to leave work now... I'm pretty sure I have it under control now, but I just don't know if I trust it...

[LeoLeonardoIII]

Any chance you can add an alert that emails your phone if an excessive number of deliverables get produced? Gotta be an error-checking layer you can add to estimate how many deliverables are scheduled and send you a message if that's exceeded by a percentage.

[GiglameshDespair]

Maybe have it cut off past a certain number? Sure, there may be outliers, but those are outliers for a reason.

[Telgin]

Any chance you can add an alert that emails your phone if an excessive number of deliverables get produced? Gotta be an error-checking layer you can add to estimate how many deliverables are scheduled and send you a message if that's exceeded by a percentage.

This is really what saved my butt.  I get an email every 2,000 or so that are sent.  After I got 2 I was curious.  When I got 5 I started freaking out.  I got about 10-12 before I finally terminated the errant scripts.

Maybe have it cut off past a certain number? Sure, there may be outliers, but those are outliers for a reason.

Amusingly this is related to what went wrong.  The entire problem was that we send text messages to customers in the form of contact lists that have to ask users to opt-in to text messages before we send anymore to them.  Some FCC law or something.  Anyway, it was taking too long when these lists were first uploaded, so we created a system to delay and throttle sending of the messages.  What went wrong was it again was taking too long and multiple copies of the server script were running (it ran periodically as a cron job), grabbing the same messages to send, and not marking them as sent before another copy of the script came along.  That gets progressively worse and worse as the server bogs down...

There were a few other incidental problems with it, but the main fix was to have the script check if another instance was running first.  That seems so blatantly obvious in hindsight, but I missed the potential problems until major problems happened...

[GiglameshDespair]

Hey, hindsight is 20/20 - it's easy to miss things that seem like they should be obvious. Look on the bright side - you won't ever forget again, eh?

[Descan]

... So you're saying your script was sending 30+ text messages asking people if they'll allow you to send more text messages?

[GiglameshDespair]

CAN WE TALK TO YOU? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE? CAN WE?