the small end of the bearings to to the center of the hub. The back bearing is the bigger of the two, and the smaller of the two bearings goes on the outside. The small end of the bearings should be pointing towards each other.
Did you make sure the rear seal is fully seated before putting the hub back on the spindle?
When I did mine I had to gently tap it into place till it was flush with the back side of the hub.
Did you replace the races (the surface that the bearings roll on in the hub)? If ya did, did you make sure they're fully seated? If I'm not mistaken, the races are pressed in, and if they aren't fully seated you won't have enough clearance to tighten the castle nut enough to insert the cotter pin.
Again, when I did mine, gentle tapping with a hammer got them installed to the proper depth.