About Store Forum Documentation Contact



Post Reply 
Uploader Memory Issue
Author Message
scotty1121 Offline
Member

Post: #1
Uploader Memory Issue
Hello! again esenthel, i last came to you with a problem regarding heightmaps and my processor, well this time it's to do with the uploader tool! When i try to upload a 1.2 GB pak file the error i get is can't allocate 123 335 345 bytes of memory.
And this is only on my computer, the memory gets out of control going up to a whopping 3.4 GB then crashing on a 1.2 gb file. Same with a 700 MB file. it just wont upload.
We tried the same settings on my team mates computer, it started uploading instantly and there were no memory leaks/errors.
Computer not overclocked at all.
Error and memory being used:
http://puu.sh/1A7xu
(This post was last modified: 12-12-2012 05:19 PM by scotty1121.)
12-12-2012 05:06 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #2
RE: Uploader Memory Issue
Hi,

Why are you trying to upload 1.2GB or 700MB files?
Are these pak files?
You should not upload paks, it is recommended to extract the files from paks, so they are uploaded separately. This way you don't force the user to redownload GB of data, when you change a single file inside a pak.

Uploader uses LZMA compression with highest compression level, which by itself may use hundreds of MB RAM just for compression.
Then uploader needs to have both original file and compressed file in the memory, which can be 1GB x2 if you have 1GB pak file.

Please extract the paks, and then upload single files.
Most likely the problem will go away.
12-12-2012 05:48 PM
Find all posts by this user Quote this message in a reply
scotty1121 Offline
Member

Post: #3
RE: Uploader Memory Issue
I was using pak files.


Also do you have any comments on my raycasting thread?, thanks.
(This post was last modified: 12-12-2012 06:18 PM by scotty1121.)
12-12-2012 06:16 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #4
RE: Uploader Memory Issue
Your program will be downloading files individually, but when using secure, they will be encrypted, so there's no worry about stealing game files.
12-12-2012 06:19 PM
Find all posts by this user Quote this message in a reply
scotty1121 Offline
Member

Post: #5
RE: Uploader Memory Issue
(12-12-2012 06:19 PM)Esenthel Wrote:  Your program will be downloading files individually, but when using secure, they will be encrypted, so there's no worry about stealing game files.

thank you very much, puts my fears at rest.
12-12-2012 06:29 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #6
RE: Uploader Memory Issue
to clarify more, files on the server will be stored compressed and encrypted, after downloading them, the patcher class will automatically decrypt them in memory, so when writing together multiple files into paks on disk you need to use pak encryption (for PakCreate/PakUpdate)
please check patcher tutorial
if you expect to download few GB of data at one time, then it's recommended to create a few paks, each sized approximately 500MB
12-14-2012 01:34 PM
Find all posts by this user Quote this message in a reply
Post Reply