BSides Entertainment System - Wearable gaming but your princess is in another castle

this talk now is about the b-sides entertainment system wearable gaming princess is in another castle and it's given by Pina so let's thank Peter for this presentation [Applause] Thank You Carlene Silvio very much appreciate the talk and I appreciate all the effort and organizing this conference that you guys have put in so my name is Peter and when I grew up I played so much Nintendo with my brothers it is insane and my parents just waited forever for us to get sick of it and I loved game boys when I was growing up and then when I went to uni I learnt about programming and software development and I hope one of the things I convey with my talk here today is when

you want to find something to work on what you want to do is you want to take something that you're really passionate about for me it's Game Boys and Nintendo and your skills which for me is software and then you can make your own awesome projects and if you if my talk inspires you to go off and break something then that's awesome anyway we'll get started okay so again my name is Peter I'm currently employed at Penn 10 again grow up with game boys if you looked at my steam accounts at the moment I haven't bought many games but I have a whole bunch of Nintendo emulators on there so I'm a bit crazy and this is my phone

that I've attached switch controllers to

did anyone notice the animation no okay just a couple of disclaimers before I get started I did not write the emulator that I used in my talk I ported it to the b-sides badge and I'll give credit to the guy who did I will not do anything illegal before after or during this talk I have no connection with Nintendo and I am NOT representing them in any way and if you choose to do some of the stuff that I've done to my badge and I'll point out the dangerous parts I'm not responsible if it stops working because I've almost had that happen a few times ok so what's up with a quick letter from the princess so

I'll go through the 2016 badge I'll talk about how I selected the emulator to use a bit about the voice so it's comparing the badge to the original nest system refactoring which was basically about getting the coats compiled in the first place hacking to actually make it function all the bugs that I found along the way and had to resolve optimizations to make it run faster and it's still not quite there yet and then how I had to reiterate again when the new 2017 badge came out oh and I got the 2018 one to work as well oh thank you okay so the 2016 badge alright so you can read up the specs online but it's a pretty

powerful little badge in fact when I saw the specs I had a dream for a long time that I want to build my own game boy and this badge looked like it was the perfect opportunity to do it had buttons you had a screen why not the Nintendo Entertainment System my first console when I was growing up it's an 8-bit system it's got a 16-bit address space which is the cartridges add to the address space about two kilobytes of RAM which is about over a million times less than your laptops most importantly it was one of the first systems to have a custom picture processing unit and this made it easier for game developers to write code

because it handled things like characters on the screen and scrolling the screen across and so it's pretty cool like some of the features I had in it before its time again why do this well I've kind of said game boys and also the nest classic is like still $300 on eBay like it's actually gone up in price I all right so I want to put a Nintendo go on it got a find in emulator so I love C++ I know it's gonna make a whole bunch of Erlang and C developers cringe in the audience sorry but so my language and the compiler Arduino look like they're all good I want to be out of C Mario so

I want a screen that works I want to definitely bad guys a little way so I want some kind of input which I never got working sorry and here their screens sound was a complication that I just didn't want to have in this project so I kind of put sound outside of the scope of this project so when you start looking for open source library online have an idea about what you want for me I wanted the emulator core that thing that emulates the system physically separated from the software that presents it and that way I can swap the view to actually be the screen on the b-sides badge and the controller is the thing that kind of joins those two

together so it tells the emulator about the screen and it tells the screen about the emulator and if you look at source files you can actually kind of get an idea about how well something's being structured so the emulator I chose looked pretty good on the surface I need to be able to compile it and if it takes me too long to figure out how to compile it I'm just going to move on to the next one like I don't really want to spend my time figuring out how compiled stuff and I can't believe it someone wrote unit tests for an emulator like that is so awesome and when you see unit tests it gives you an idea about

the quality of the products that you're actually getting as well and so this guy Darrell you it was his emulator that I poured it there's a github link address there so if you're in the audience thank you very much happy to talk to you afterwards all right so let's do a bit of a comparison [Music] the original nest ran in about two megahertz it's about and then the badge itself is at 80 so 40 times faster and I still can't quite get it playable but when I was looking at it four times faster that's a pretty good ratio to do an emulator color screen looks great no worries that should be easy to do memory

it's got about 90 kilobytes of memory on the badge and I probably only need about 32 to actually make it work so enough memory storage it's got one megabyte of flash on the badge put it in perspective a three and a half floppy disk had one point four megabytes so it's a less but we should be able to fit it all on there now I looked into being able to load games from sd card but given the way the badge is designed I would have to share about the bus to the SD card with the screen which would slow down the graphics so I thought rather than doing that and risking slowing down the device

any further I would just burn the roms into the program and then flash the whole thing onto the badge so it only runs the game at a time and when I say slow I mean like the Star Wars opening credits scrolling slow so I never got to control to work just didn't really have time at the time when I was doing it this is a schematic for the original Ness controllers and how they work if you're curious look it up online it's pretty simple the way that it does a latching system anyway if you're a nerd like me may be worth the read so now we have an emulator and we have a device and we need it to compile we need the

code to be compatible to be flashed on through the device so this is what I thought I got this is why was hoping for this is what actually ended up happening under the hood where the emulator the graphics it all knew about each other so the first part of refactoring was actually just separating out the code so the emulator had no knowledge of what the graphics system was and when I say graphics system I mean the Windows graphics system or the Arduino TFT library and this is what I ended up refactoring it to so have my main program the emulator and some DirectX graphics stuff which is great because I can keep the same emulator and swap it

to Arduino and use the screen so I can actually test test my code on a Windows PC which is far faster than flashing every time I want to try something and then when it's good to go I just hit recompile for Arduino and we're good to go so storage wise I'm only using about 62% of the one megabyte on the device so storage I got plenty but ran with a different issue so my laptop has like 8 gigabytes of RAM which is like millions times more than what's on this device and I got out of memory warnings when I was compiling it so what I did is I moved all the memory allocations in the

emulator right up to the top of the program and then handed it in and that allowed me to measure how much memory was being used by each component and on my first cart I needed it about 400 to bytes of RAM on a 90 kilobyte system so it was a bit depressing at this point because it looks like it's not going to work so what do you do we start hacking because that's what we're all here to do right we're going to fine tune it until the square peg fits in the round hole so the biggest culprit is the screen you've got 256 like 240 by 4 bytes per pixel so that's already about 250 kilobytes of

memory but we need to basically shrink down so the first thing to know is that the emulator was using 32-bit color but most of your phones and screens only use 16 it's an 8 bit system nobody's going to know the difference so we'll just half the amount of bytes needed per pixel and we can have the storage done so then I found out that the actual LCD screen has about 1/4 of the pixels of the original Ness how how did you do it Sylvio how did you pick a screen that has lower resolution than the original Nintendo Entertainment System but turns out you can use this your advantage because if you only have a quarter of the pixels you only need a

quarter of the storage so it was like a bonus in the end so all I did was every time I had a screen I added up 4 pixels took the average and said that's the color and then it's like well why are they still doing a whole picture of rendering at a time the original Ness worked by drawing a line on your old CRT at a time so looking a bit deeper in the emulator I found the piece of code where it was basically accumulating lines storing in a picture and then handing it back and I hooked it earlier so now I only need one row of pixels at a time and now we've gone from 250 kilobytes of

storage of RAM sorry to 1 kilobyte which is pretty good I thought and then we had brown so the actual RAM is like a pre-allocated buffer for the nests and all the games get loaded into it there's no way that I can sacrifice 65 kilobytes of storage so I ended up reading the game as it would in the actual system directly off the flash rather than loading into memory but I'll get it bit more into that later on so I finally got it down to about 87 percent of memory usage and at this point you're stuck being warned by the compiler saying hey look if you want to flash this on your device you go right ahead

but I think there could be problems and then obviously things don't necessarily work straight away when you try and recompile it so it's a constant cycle of compiling fixing compiler bugs going back and piling again over and over and over again and then you get to the fun part where it finally flashed them to the device and it doesn't work and you get the blank screen again of death so I don't know much about the internals of the emulator so where do you start there's this idea and programming of Indianness and if you want to think of it when you go to the shops and you buy a slab of beer and you prune next to

your fridge at home and you're taking beer cans out of the box and put in your fridge little endian is about whether you stack the shelves in the fridge from left to right or right to left and so given that my Windows machine stacked it left to right and my badge stacks it right to left I just looked through the code for all places where it was doing this behavior and added some extra code checks so then it actually started running and I got a stack trace and as we know didn't really have a debugger that I could configure at the time so I basically added a whole bunch of print statements to the until I found out the exact line turns

out we thirds we know there's a different memory copy function called memory copy P and that's what you use to basically take from flash and put it into memory and then I got my first every screen I'm like yes it's working but it's blue and I had gone through all these little endian cases but it turns out that because I knew it was the color all that was is that think about it being red the other way around the blue pixels being red is red and vice versa so look through the code for all the pixel handling found the spot and it's still crashed so what I was doing is I had the emulator running

inside one loop of the device and this is concept with embedded devices of a watchdog timer and you are responsible for calling the watchdog timer reset because if it doesn't happen it will reset the device for you and that prevents your device basically hurting itself when it gets into a bad state so every time the emulator rendered a frame I just called watchdog timer reset and it was all good so my dog is getting a walk roughly 30 times a second so she's pretty happy oh and that's my dog ADA she's pretty cute and we got it working

[Applause] but it was really slow so now we're gonna find all the ways to make it run faster first easy thing we're just gonna have the frame rate now I know if the game is in here that are operating at 144 Hertz 30 frames per second it's probably gonna hurt your head but you got to do what you got to do but it's not as bad as trying to do interlacing we're basically on odd and even frames you decide to render odd and even lines and it looks terrible in fact if you try and like back up a DVD some of them are actually interlaced and you get the same kind of effect when you play back the

backups so the other thing is that the screen doesn't change too much on the old systems so you can actually add up all the pixels in a line and create what's called a checksum and then only choose to draw a line when the checksum changes so this added a fair bit more speed although it adds speed depending on what's happening on the screen so it has side effects as well and then you can actually rather than sending it a pixel at a time if you know that you're going to have to render five red three blue one red couple of green you can actually tell the LCD that you can say I want five red instead of saying I want

to red red red red and so it actually makes it run a bit faster you can also overclock it so on your badges you can actually call this function and make it run at 160 megahertz which again I don't know how long my badges are gonna last being over clots and then you can actually overclock the SPI bus as it goes to the screen now I don't know whether you can see it when I put up to 40 megahertz what happened was is that the data wasn't being sent to the screen so fast it could actually catch all the pixels and so you start to see little imperfections on the screen because it's being sent to quickly

so I took it down to about 25 and then I thought well what if instead of reading from the flash every time the game wants to do something I actually implement a paging system and I never quite got it working it crashed a lot but the few times it actually did run through it was like playable kind of fast so a little bit more to work yeah and I think it would be like almost usable so still work in progress and then came the 2017 badge but lucky for me sílvio tweeted his workshop and I don't know whether you guys can see on this picture but the actual badges were on a tomb and you could actually see the

pin outs which basically indicated to me that he was going to use the exact same microcontroller and also there's a row of pins on the top which suggests he's going to use the same screen so when I got my badge I thought no worries I'll just flush it and it didn't work so as I okay well I fixed a whole bunch of problems it's got to be something to do with the wiring now normally if you're doing PCB work you have a schematic to work from I didn't have that if you actually do things the right way you can use a multimeter to actually see where things are connected I didn't think to do that I trace the circuits on the PCB

one by one and figured out by reverse engineering for wiring for the LCD for which two wires were around the wrong way I highly recommend the multimeter approach it would be far quicker and frustrating than what I did and then it kind of works but the colors are wrong and it's around the wrong way and I looked at forum after forum online trying to find someone trying different things I think that's a big important thing when you're doing your own project you're going to be prepared to try a whole bunch of stuff that might break what you're working on but then get up and running again so that was pretty good I was pretty happy

[Music] that's about it any questions oh yeah see us coming up

maybe I think here how hard was it to get the 2018 batch going so I could you repeat the question sorry how hard was it to get the 2018 badge working the question was how hard was it to get the 2018 badge working besides me trying to wire it up and sort of see how it worked it wasn't too bad I think my soldering and my shaky hands was the biggest barrier to what I was doing but essentially what I ended up doing was wiring the screen from the 2017 badge so exactly the same pins on the micro so the code just worked so I took the easy way in that respect are there any more

questions what over here did you consider any other hardware modifications to make changes to the software easier like adding more ram or something okay the question was did I try any hardware modifications to make it run faster or make it easier it was actually suggested to me later last year and I need to actually get around to it but absolutely adding some more EEPROM sorry some more round to the side of it would have made some huge improvements I did just get working before I came here a week classic controller working you can get little PCBs that click on the bottom of it I don't actually have one he unfortunately was in a bit of a rush

this morning but um yeah you can actually get nest classic controllers to work with the badge it's basically just an I squared C bus so but that was really the only other hardware modification I tried anymore let's think hey thank you very much [Applause]