Microsoft | Learning FAST University | FSIS Tech Corner: Memory Allocation

Microsoft | Learning FAST University | FSIS Tech Corner: Memory Allocation

Benjianna_Schomer

I am often asked why the default memory allocations settings are so low for the FSIS Server. The answer is simple: the default settings allow all the FSIS components, including the FSIS server, ESP search engine and connectors, and Visual Studio, to run on a single machine with 8 GB of memory. For the FSIS classes the VM is actually only allocated 6.5 GB of memory, because the flows used in the labs are simple and feeding batches are small.

Production
The default and recommended amount of memory reserved for the different nodes is as follows:

  • Administration node: The default setting is 512 MB and is sufficient for most environments, even with a full allocation of worker services.
  • Content Management Service (CMS) node: The default setting is 800 MB which can only run the simplest flows. In production, memory should be increased to 4 GB. To better utilize memory and CPU, add additional CMS services.
  • Interaction Management Services (IMS) node: The default setting is 800 MB, which is sufficient for simple flows with a single search result. When federating results from multiple engines, additional memory is required to hold and process the results. As a rule of thumb, each additional federated source or thread operator in your flow increases the memory allocation by 400 to 800 MB.
  • Content Distributor Emulator (CDE) node: The default setting is 400 MB, which is sufficient for most operations. The amount of memory must be larger than the expected maximum batch size. If not, either increase the memory allocation to hold your batch or decrease the batch size. |\

Development
I am fortunate to have 16 GBs on my development system, so I can allocate plenty of memory to FSIS and ESP. If you do not have a PC with 16 GBs, here are a few tricks to improve performance when running everything on one machine:

  • Double the memory allocation for CMS to 1.6 GB. I have been successful doing this on an 8 GB machine, but not with a 6.5 GB VM.
  • Disable the enterprise crawler. If you need to feed in web content use the Docpush command line tool instead. Usually when I am testing CTS flows or document processing pipelines I only need to feed one document at a time.
  • Reduce your batch size when feeding with FileTraverser, JDBC or other connectors. When ESP and FSIS are running on the same machine each byte that the batch size is reduced, reduces the amount of memory used by 3 bytes, because the batch is created in memory on the connector, stored in memory by the content distributor and processed in memory by the document processor. The connectors and content distributor will grab as much memory as they can, up to 3 GB.
  • When testing and debugging CTS flows, disable the QR Servers to free up memory. I disable the QR Server when I am feeding. When I need to test if the documents are searchable, I disable document processing and re-enable the QR Server (psctrl stop and nctrl start qrserver).

On occasion I have had to work with 4 GB machines. In that case, the only option was to put FSIS Server and ESP on different machines. For more information on deployment models in production or development and the procedure for updating memory allocations attend the Microsoft | Learning  FAST University instructor-led course FAST Search for Internet Sites for IT Professionals. To register online visit www.fastuniversity.com or contact an Education Consultant for assistance at fastuniv@microsoft.com.

By Brian Barry

Comments
  • rellufgerg
    |

    Not that such an addition will help everyone's computer.  But if your computer seems to be dragging, by far the best combo of improving performance, low cost and needed techie skill involves increasing one's computer memory.

    In essence, memory does not make your processing faster -- it increases the number of things your computer does at one time.  If you REALLY do only one thing at a time on a computer -- closing each program before opening the next -- you need professional counseling beyond my pay grade.