Yes, you can run multiple Sybase instances on the same machine. On Windows this is easy as long as you're using the same ASE version (it gets a bit more complicated if trying to run different versions of ASE on the same Windows machine).
As for why you're having problems ... *shrug* ... you would need to post more details since there's no way to tell (from what you've posted) if you're having Windows problems, Sybase ASE problems, SAP application/installation problems ...
--------------
If you plan on posting a large volume of log file details I'd recommend you use the advanced editor (shows up as a link in your reply window) so that you can attach the log file details as an attachment to your posting.
--------------
While there's no problem with posting in this forum, I'd recommend you consider posting in the SAP Applications on SAP Adaptive Server Enterprise (SAP ASE) forum. That forum is geared specifically towards SAP applications running on ASE, ie, you'll likely get the attention of more folks who can help with SAP/Sybase ASE issues.