Despite generally resisting the hype train, in the past year I've jumped onto two relatively new Facebook projects. ReactJS and HHVM. I'm a huge proponent of both and enjoy working with them immensely. Except when I don't.
Around 3 months ago I chose HHVM for a new project and was initially very excited by the static analysis tools, built in profiling and great performance. Despite manygreatexamples of HHVM deployments, shortly after deploying to production we started hitting OOMs pretty frequently:
OOM killing our HHVM server
With some heap profiling I first located a leak in mongofill-hhvm, and removed our one instance of create_function (known to leak memory) but still experienced poor reliability post-fix. At this point, we jumped onto the nightly builds as some leaks were fixed. At this point, we started hitting segfaults and our HHVM binary occasionally refusing to accept fastcgi requests.
Coupled with some nginx load balancing, that gave us an instant noticeable boost in stability.
upstream hhvm { server 127.0.0.1:9010; server 127.0.0.1:9020; server 127.0.0.1:9030; server 127.0.0.1:9040; server 127.0.0.1:9050; }
With my fast approaching Europe trip, I didn't want to be tethered to my terminal and wanted it to detect failure automatically. nginx does provide health_check in its commercial builds, but at this time I'm not too inclined to pay $1500/server/year for the pleasure. Thus, I needed to ping these with minimal effort. I decided to go directly to the fcgi process rather than use nginx to avoid needing to work around the load balancing, and remove the web server as a potential false positive.
I considered learning the fcgi spec but decided to cheat instead. tcpdump to the rescue!
A tcpdump of HHVM requests
We only really care about the 32 bytes of data used in the first request (indented for emphasis), which cleaned up a little look just like this:
supervisorctl status | awk '{print $1 " " $4 " " $6}' | tr -d ',' | while read NAME PID UPTIME; do PORT=${NAME##*_}
MATCHES=$(echo $PAYLOAD | tr -d ' ' | xxd -r -p | nc -w 3 $HOST $PORT | grep 'X-Powered-By: HHVM' | wc -l) if [ $MATCHES -ne 1 ] then logger -s "$NAME is not functioning as expected, restarting..." supervisorctl restart $NAME fi done
Since implementation, we've had no observed instability and can sleep a little easier. Next, we will likely implement ngx.location.capture_multi on our read requests so that one HHVM node can fail without any end-user impact.
I'm incredibly grateful for the work the HHVM team has put in, and fully intend to stick with it despite the bumps. Hats off to all those involved and the incredibly active community!
Unfortunately, I was wrong. So I contacted their support team to see what's up!
Thanks for the help, Brian. /s
Being unaccustomed to "no", and particularly impatient I decided to develop my own solution utilizing the very same API's Shopify created for themselves in their admin panel.
So it all began when I decided to poke at how they're being loaded into Shopify's back-end
GET /admin/discounts.json?limit=50&order=id+DESC&direction=next HTTP/1.1
Nothing particularly special, there's a token we need to watch out for and, of course, our session cookies. So first thing's first, let's tackle the login form. I've trimmed this down to the bare necessities for your viewing pleasure
Spoiler: Looks like Shopify are at least playing with OpenID integration
In the interest of minimizing maintainance let's parse all of those <input>'s dynamically. I chose to use regex over a DOM parser because it seemed more appropriate in such a hacky project, and will save us having to worry about broken markup. We'll do this in two parts, first we'll grab the login form, and then the key/value pairs embedded within it.
The astute among you may notice the setToken call at the end. We'll get to this shortly. Also, setOptions is a function I crafted to keep the code clean, it will take care of setting the cookie jar and user-agent upon each request. Yes --Cake, and cookies.
Once again we'll resort to regex. By co-incidence, or not, the arrangement of these parameters has switched once before, so that's worth keeping an eye out for!
Sure enough, that worked too! So what other cool stuff can we do?
Spoiler: I like graphs.
You may have noticed the flashy new dashboard in Shopify 2. Fortunately, with little effort, we can access this data too!
There's a couple of things we need to take note of here, the callback (this is JSONP, we'll get to that in a moment), and the token. The token is used as authentication, and set inline in the document body.
As soon as I saw this it was pretty obvious it was a base64 string and an md5 hash, whilst this probably isn't terribly useful for us it's nice to know! Let's decode it. (Note: I snipped these to keep this store private)
["1989xxx", "2013-03-07T22:34:12-08:00"]
So the base64 is an array containing the store ID and a timestamp. Perhaps the hash is used for performance metrics, or more likely a checksum of the array to avoid people grabbing analytics of other stores. Doesn't matter much to us, as we aren't trying to do anything malicious here.
Due to the same origin policy XHR requests to external locations (scheme, hostname and ports must be consistent). The exceptions being JSONP, and CORS. CORS is considered a better solution however in this instance Shopify is using JSONP, that's what the callback parameter is for. We'll need to strip out that callback when we parse the response.
To do so, I've defined the callback as a static fake_function and strip it out with regular string functions:
The duplication of items has been a huge issue in a variety of games, ruining the economy in both official and in particular, private servers. A select few have figured out solutions to this,
however the majority of which are offline solutions usually within stored procedures which hasn't proven too reliable since the database is only amended every 5 minutes by default in Knight Online.
As such, we needed to figure out a live way to do it.
Fortunately, to communicate between the various server files a block of shared memory (or Memory Mapped File) is used. This means tapping into the data isn't all that difficult.
Reversing the structure is much trickier. As such, I'll provide a near-complete structure below which will serve as a good base for this article.
BYTE m_bNation; BYTE m_bRace; short m_sClass; BYTE m_bHairColor; BYTE m_bRank; BYTE m_bTitle; BYTE m_bLevel; int m_iExp; int m_iLoyalty; BYTE m_bFace; BYTE m_bCity; short m_bKnights; short m_sClan; short m_sUnknown; BYTE m_bFame;
We will cast the block of memory to that structure so we don't have to loop, and apply lots of mathematical offsets, it generally makes the code cleaner to work with, and easier to maintain.
So first of all, let's connect to the file and build up our local array. (You'll probably want to place this in your main, or equivalent.)
Note: I'm not going to incorporate lots of error handling etc in this guide, you will need to do that yourself.
Okay, so this will open up the shared memory files that Knight Online uses to access the user data from all the various applications (aujard and ebenezer for example) and map it so we can access its data from our application. Next up, loading all of the users from the memory block and casting it to a nice struct which was defined above!
That's a rough outline for our function, we use a mutex to prevent any of our other functions accessing the array while it's being written to which could cause memory exceptions and such. So let's load our users now.
for (int i=0; i < MAX_USER; i++) { pUser = (_USER_DATA*)(m_lpMMFile+(i*8000)); currentUsers.push_back(pUser); }
ReleaseMutex(m_hUsersMutex); }
Because the struct above isn't entirely complete, when looping each user we can't use sizeof(_USER_DATA) as we ideally should - as such it's hard coded as 8000 bytes, this may need maintaining in future versions.
This will build an array of pointers to each user - so all data is always accurate without you needing to re-populate. This has its merits and cons, so in some situations we need to create a static array too which we'll do next in the dupescanner example.
So next, let's just take a look at iterating through the array so we can actually use it!
std::vector<_USER_DATA*>::iterator iter; for (iter = currentUsers.begin(); iter != currentUsers.end(); iter++) { // our magic }
Really simple stuff there, you can then access the properties of each user directly. Beware of the null structs (as again, these are simply pointers to a block of memory) so do a check for something like (*iter)->m_id having a positive length.
An example of applying the above code could be to find a pointer to a struct from a given username, how about this:
std::vector<_USER_DATA*>::iterator iter; for (iter = currentUsers.begin(); iter != currentUsers.end(); iter++) { if (!_strnicmp(user, (*iter)->m_id, MAX_USER_ID_SIZE) return *iter }
ReleaseMutex(m_hUsersMutex); return FALSE; }
There are so many more potential applications to this code - for example a decent speed hacking detector, zone scanning for if users are stuck in the war zone, or bugs with invading and whatever, and of course the biggie, dupe scanning.
Let's create a quick draft on how you may achieve something like this!
You'll probably want to create it in a separate thread, so add something like this to our definitions
for (iter = pTemp.begin(); iter != pTemp.end(); iter++) { _USER_DATA* pUser = *iter; if (!strlen(pUser->m_id)) continue;
for (int i=0; i<HAVE_MAX+SLOT_MAX;i++) { if (pUser->m_sItemArray[i].nSerialNum > 0 && pUser->m_sItemArray[i].nNum > 0) { if (!userMap.insert(make_pair(pUser->m_sItemArray[i].nSerialNum, *pUser)).second) { _USER_DATA* otherDuper = userMap.find(pUser->m_sItemArray[i].nSerialNum)->second; // Do some magic } } } } }
You may notice that I only ran it once in the above example, you may want to add an extra parameter for bEnabled, then add a while (bEnabled) { sleep(sensible integer); pTemp.clear(), re fill, the interation, etc } to run it constantly!
I haven't gone into the full depths of utilising the mapped file to send packets (for disconnecting and such) in this guide it's focussed KNIGHT_DB only, but I may cover that at some point in the future.
To initialise the thread, it'll be something like this:
I was continuously asked how to apply security patches which were being released,
instead of including instructions in each patch that I released I decided to put together a quick general guide.
While this guide primarily focuses on Knight Online private server files,
it serves as a good example and all terminology and methods will apply.
To begin, we'll start with the screen you'll be prompted with just after you open ebenezer in OllyDbg. I'm using Ollydbg because it's easier, don't use a hex editor it's just silly. Olly will translate opcodes to hex for me and will make far less mistakes!
Also note, I'm using the classic version (v1.10 final) - the v2 alpha is becoming increasingly popular and sports some exciting new features and some amazing improvements. So be sure to check it out! The images now reflect OllyDBG 2.01
On the right, you have the first screen you'll be confronted with after loading your executable. Briefly, starting from the top left you have your 'CPU', this is the main window you'll be using which consists of a disassembly of the image from memory at the specified location, this is what translates your 'hex' into assembly and visa versa.
To the right hand side you have your registers, you may think of these like your typical variables but remember they're pointers, either way these represent the values your assembly code is using.
Just like in most native languages you need to de-reference a pointer before you use the value, this is why you may see "DWORD PTR DS:[pointer]", which means there's 4 bytes at this location that we want, and we want an unsigned value of it. For all you fellow readers from a linux background, a DWORD is typedef'd in the winapi as an unsigned integer.
On the bottom left we have the dump, this is usually where I'd follow the values of registers but it's multi-purpose and is also very handy for viewing a region of memory, think of this as your traditional hex editor.
Finally, to the right of the dump is the stack, I wont fully explain it here since it's not necessary for this patch, generally you can use it for following parameters passed to a function and it's tempory storage. Every thread has its own stack.
Now, moving onto the actual patch. Let's navigate to the area we want to patch, on the far left of the CPU you have the address you're currently viewing. The currently executing line of code is at eip (Instruction pointer, a register).
We want to goto 0x00498B59 (The 0x represents hex, like &H in basic).
To do this we press CTRL+G or can navigate by the menu, a window like this will appear:
Press enter (or "Follow Expression"), and it'll take us to the code we want to modify. As you can see, it's exactly how osmanx said it would be. Now lets make the appropriate adjustments. You can see here that it's comparing the 8bit BL register to 0x11 (Rememer everything is in hex). If it's equal, it goes to the same location as if it's 0x07, This looks like it was meant to be expanded at some point. They're both going to the same location regardless, so we want to change 0x11 to goto our codecave.
To do this, double click the line we want to modify which is 0x00498B5C and modify it to the code displayed. "JE 00499218"
Press "Assemble", if all goes well and you typed it correctly it'll modify the hex appropriately for you.
Now, tap enter to follow that jump to our code cave, or use CTRL+G -> "00499218" if you want to do it manually.
This will take you to our code cave which is just a block of INT3's, this instruction is used to trigger a breakpoint but is also often padded between functions. It's an ideal location for us to make our patch since it isn't executed!
Assemble, just like you did before our block of code.
Next, right click in the main CPU window, navigate to "copy to executable" and select "copy all modifications"
as shown in the following image.
Then it'll copy the memory image to a new window and allow you to save it to disk with the modifications, you can do that once again by following the navigation to "Save File" as shown to the right
OllyDBG 2.0 will give an additional confirmation menu
Then a little window will appear asking you where to save it. I'd suggest you make a backup EVERY time, incase you make a mistake or the patch is incorrect (though i'm sure it isn't!).
That concludes this article. I hope I didn't miss anything important out. If you have any questions feel free to e-mail me I feel as I fear I may have rushed towards the end but I will amend it appropriately if anyone points out a mistake or particularly vague area.
Recently a new vulnerability has been exposed (and patched) which targets specific platforms as a result of certain assumptions made within the zend core (specifically zend_strtod.c) when handling floating-point arithmetic.
The issue was only being manifested by gcc builds with -O2 so a recompile could fix it, reportedly -O0 fixes it but I would recommend using -mfpmath=sse which will favour a newer instruction set rather than the older deprecated x87 math instructions. A diff of the new PHP revision revealed the actual commited patch was an additional keyword in a declaration:
double aadj, aadj1, adj;
vs
volatile double aadj, aadj1, adj;
The volatile keyword instructs the compiler not to perform optimisations
Fortunately this is an x64 server and doesn't adopt the x87 fpu but it did affect my laptop. I applied a quick software patch which was something like this (Note: both must be placed at the top of your index, or executed file):
if (strpos(implode($_REQUEST), '2.2250738585072011') die();
or
array_walk($_REQUEST, function (&$x, $v, $k) { if (strpos($v.$k, '2.2250738585072011')) unset($x); });
The first example will stop script execution if the dangerous value is detected, the second will unset the affecting variable - with improper error handling in some situations this could cause errors, though.
Fortunately however, as previously stated a patch has been commited so if possible upgrade to PHP 5.3.5 or 5.2.17
