1	Linux on z/VM Configuration Guidelines Barton@VelocitySoftware.com HTTP://VelocitySoftware.com HTTP://LinuxVM.com
2	Overview Configuring z/VM for Linux on zSeries Must configure z/VM – many defaults incorrect Linux must be configured for shared resource environment Many actions not intuitive Infrastructure unknowns for “new” installations How to manage performance / capacity planning? What are the limits of a configuration and how to measure How to share resources to reduce ROI . Measurement and Tuning for z/VM IS Required Start with Proper Configurations
3	Configuration Options General Storage Options Linux Options Storage Sizes Swapping for Linux Linux virtual processors Network z/VM Configuration I/O, FTP Topics MDC Paging and Spooling for z/VM DASD/Cache/Channels z/VM System parameters Expanded Storage Infrastructure Linux infrastructure – monitoring availability and performance
4	General Storage Requirements Configuration requirements different for Small Infrastructure Servers – “Small” Systems DNS, Apache, Samba Low I/O rate Real storage less than 2gb Virtual servers sized 64mb to 256mb Medium (31bit) Application Servers - Small to Large Systems Websphere, Domino, Oracle z/VM Real storage greater than 2GB High I/O rate potential Typical 512MB to 2GB Large (64bit) Application Servers - Large Systems Oracle, SAP z/VM Real storage greater than 10GB High I/O rate potential Virtual Servers Typical 512MB to 16GB
5	Storage Considerations z/VM is shared resource environment Over-committing storage improves costs per server Over-allocating storage reduces servers that can be supported QDROP IS QUITE IMPORTANT Storage requirements of Linux very high Linux designed for dedicated storage, references all storage Linux is LRU, competing with VM’s reference pattern High percent of referenced pages – what can z/VM page out? Linux does not drop from queue – 100 timer pops per second was 1^st problem, fixed. CP storage management bypassed, forces “emergency scan” Current release of IBM JDK (WAS) polls 10 ms fixed in 1.5 SR6. Upgrade to SR6, use: -XsamplingThreadExpirationTime=0 or IBM_JAVA_OPTIONS=-Xjit:samplingThreadExpirationTime=0
6	Qdrop issues Storage in use by LINUX07 has 166K pages Server was active 8 hours prior, but idle for 8 hours never dropped from queue, never gave up storage Active server LINUX02 must compete for reduced storage Guideline: Force Linux Servers to drop from queue
7	z/VM Storage Architecture Issues 64-bit not complete in z/VM 5.1 All I/O buffers must be moved to the z/VM real 2GB storage Impacts “application servers” doing I/O SLES 9 has new DASD Driver that bypasses problem Storage architecture problem symptoms: Slow response with no obvious reason, correlates to I/O workload High page rate even when storage over configured Monitor is inconsistent Impacts database, fileservers, “tar” Problems usually caused by one or two servers Typically the “large (64-bit) application servers” are the problem Many “medium (31-bit) application servers” can cause the problem Move problematic servers to another LPAR Reduce virtual machine size of problematic servers GO TO Z/VM 5.3
8	Linux Storage z/VM Paging Over commitment of storage causes paging Over commitment of storage reduces cost Paging is common (manageable) performance problem Linux Swapping Swapping result of over commitment of Linux storage Swapping to vdisk very fast, uses storage when it happens Swapping to dasd very slow, always noticeable
9	Linux Storage Linux Cache Linux avoids I/O by using cache Linux will cache gigabytes of data if allowed Oracle SGA MUST fit in cache Use this feature in z/VM environment Swap historically was slow SCSI device Reduce size of Linux Virtual Machine MAJOR Knob. Reducing virtual machine size reduces caching of old data Define virtual disk for swap Virtual Disk paged out when not in use - Unlike “Real” memory Experiment with Linux server swapped 40,000 per second.
10	Tailoring Linux Storage
11	Linux Swapping Reducing virtual storage size may cause swap Linux does not swap until out of storage Swapping to disk VERY VERY SLOW Other platforms increase storage size because disk is slow Swap to disk if you want to penalize a server Max swap rate maybe 200 on a very good day Linux Swapping to Vdisk Very Very FAST Swap rate experiment swapped 40,000 / second
12	Linux Swap to Virtual Disk Virtual disk for swap pros Very very fast swapping Very high bandwidth Requires very little resource if unused Swap I/O does not require below the 2gb line storage Virtual disk cons Requires control block storage below the line Swapping requires CPU, must manage swap rate Swap Guideline: Define 2 virtual disks, prioritized swap Use DIAG driver instead of FBA - Reduces I/O by factor of 8
13	VM Storage Overview, Paging Hierarchy
14	z/Linux Page / Swap Hierarchy
15	z/VM Paging Hierarchy
16	Linux Storage Case Study First case study: Process took hours, system paged significantly Reduced size of Linux Virtual Machine, 128mb to 24mb Defined 100MB Swap disk Linux reduces storage requirement Process took minutes Virtual Disk paged out when not in use This works!!! Paging greatly reduced, Linux performance greatly improved!!! This research critical to using Collaborative Memory Mgmt (CMM)
17	LINUX Swapping to VDISK Change 128MB Server to 24MB with 100MB Swap Reduction of Overall Storage Requirements of 100MB Unused VDISK is paged out
18	Virtual Storage vs Virtual Disk tradeoffs Virtual Disk I/O 838K / 900 seconds About 900 - 1,000 per second (NOTE MDISK HIT RATE!!!!)
19	Cost of Swap
20	Virtual Disk Analysis Report: ESAVDSK VDISK Analysis Report Linux Test ------------------------------------------------------------------------------ Maximum VDISK: Blocks (MB) System storage: 157M 76800 Storage per user: 8389K 4096 <--Size---> <AddSpce> Priv VIO pages AddSpc VDSK Cre- Del- or rate User Resi- Owner Space Name Pages Blks ates etes Shrd /sec Links dent -------- ------------------------ ----- ----- ---- ---- ---- ---- ----- ----- 14:02:00 PAZXXQ01 VDISK$PAZXXQ01$0203$0010 25856 205K 0 0 Shrd 0 1 32 PAZXXQ01 VDISK$PAZXXQ01$0501$000F 25856 205K 0 0 Shrd 0 1 32 PAZXXQ02 VDISK$PAZXXQ02$0201$0011 25856 205K 0 0 Shrd 0 1 32 PAZXXQ02 VDISK$PAZXXQ02$0203$0013 25856 205K 0 0 Shrd 0 1 32 PAZXXQ02 VDISK$PAZXXQ02$0501$0012 25856 205K 0 0 Shrd 0 1 32 PAZXXT03 VDISK$PAZXXT03$0202$0037 524K 4194K 0 0 Shrd 414 1 523K PMZXXQ01 VDISK$PMZXXQ01$0207$0033 524K 4194K 0 0 Priv 0.02 1 4210 PMZXXQ01 VDISK$PMZXXQ01$0208$0034 524K 4194K 0 0 Priv 0 1 542 PMZXXQ02 VDISK$PMZXXQ02$0207$0035 524K 4194K 0 0 Priv 0 1 543 PMZXXQ02 VDISK$PMZXXQ02$0208$0036 524K 4194K 0 0 Priv 0 1 541
21	Additional Storage Performance
22	How many Virtual Processors?
23	How many Virtual Processors?
24	Network Storage Impact OSA Adapters: Dedicated to Linux servers or virtual routers Require approximately 8MB of 2GB real storage per adapter z/VM 4.2, 4.3, 4.4 – no qdrop z/VM 4.2 Use Guest Lan – Reduce OSA Adapters, Requires virtual router (VM TCPIP or Linux) z/VM 4.4 Use Virtual Switch - Eliminate virtual router z/VM 5.1 Qdrop occurs with dedicated OSA Guideline: Use Virtual Switch, Guest LAN
25	Mainframe I/O Expectation Issues Mainframe I/O expectations OFTEN wrong I/O traditionally tuned to operate within limitations Separate I/O processors Competition not limited by ESCON channel speeds Customer says “FTP on Linux under z/VM is slow” Benchmark was large FTP, problem NOT network Escon channels, 30ms “CONNECT” time 500K transfers Questions: How fast are ESCON channels? FICON channels? Ficon Express? How fast are SCSI disks on other platforms? PAV? What are options when high utilization on shared disks? PAV Available z/VM 5.2 - Use for high activity shared devices ONLY
26	FTP Benchmarks: Results NOT intuitive
27	FTP Benchmarks: Results NOT intuitive
28	Benefit of MDC
29	MDC Pains
30	z/VM Page Space Overcommitting real storage is good, reduces cost Back up is Paging storage If 40GB main storage Overcommit factor of 2 - How much paging storage needed? VM installations often very underconfigured Guideline: Paging storage should still be 2 times requirement Number of paging devices? Number of channels? ROT not valid Lack of page space planning is top reason for first installation z/VM outage
31	Expanded Storage Expanded Storage required for paging performance True LRU Page the correct pages Page rates to disk drop when converting real storage only system to real+expanded How much expanded? Enough for 30 second window Enough so STEAL does not page to disk 20% usually enough Measure on ESABLKP
32	Infrastructure impact on CPU Shared resource environment: Avoid unnecessary work Avoid “waking up Linux” Availability Monitoring – necessary? Using Encryption - necessary if on virtual lan? Measure your infrastructure and determine scalability!
33	Infrastructure: SOP Valid? Question: Why always hit every 15 minutes? SOP: Standard Operating Procedure
34	Infrastructure: Alerts Show process by ID Status Total CPU Percent CPU Storage (Non-velocity mib)
35	Performance Instrumentation Performance Instrumentation Cost of instrumentation often excessive “Native Linux” tools will not detect many problems Agents may take 5-10% of a processor Cost of instrumentation should be < .1% per server Performance instrumentation should not change performance Active agents vs Passive agents Active agent wakes up at constant interval and records data Passive agent only responds to external request Dynamically turn off monitoring of idle servers!!!! If z/VM data shows server is idle, should agent wake up to find out what is running? Logon to a server to tell the agent to stop working?
36	Agent (SNMP) Overview Simple Network Management Protocol “Passive agent” TCPIP application usually provided by TCPIP Vendor Requires operation of the SNMP Daemon NET-SNMP is SNMP implementation for Linux Available on WWW.SourceForge.NET (net-snmp project) Supports Linux, Solaris, NT, HP-UX, others… Standard on SUSE and REDHAT TOP, other agents “lie” when under z/VM Sample of factor of 10: HTTP://velocitysoftware.com/present/CaseAFS/ The new “monitor interface” does not help Misleading record update for CPU data makes problem worse NETSNMP validated against top Processor, Storage, Processes: HTTP://velocitysoftware.com/present/TOPvSNMP/ Possible to prorate linux process data against z/VM monitor data
37	Linux Configuration Summary Virtual machine size Minimize until some swap Swapping Swap to virtual disk Define 2 virtual disks, One to meet the average requirement Second one for overflow Use DIAG driver instead of FBA Reduces I/O by factor of 8 Virtual processors Minimize to meet the workload/application requirement Infrastructure costs Minimize – shared resource architecture
38	z/VM Subsystem Configuration DASD Channels ESCON channels are 17MByte / second Ficon channels 100MB, Ficon Express 200MB Ficon compares to SCSI disks on other platforms Paging/Spooling How much spooling is required to dump 40GB server? How much paging is required to support 2 times over commitment of 40GB z/VM system? MDC Caches data – read-ahead, often used data Does not require 2GB storage for I/O Default too high SET MDC STORAGE 0M 128M SET MDC XSTORE 0M 0M
39	z/VM Expanded Storage Expanded storage to z/VM is like vdisk swap for Linux 2GB issues results in paging even when high storage available Expanded storage is very necessary for paging hierarchy Expanded Storage Requirement: 25% if using “small infrastructure servers” More if using “large applications servers”: If paging to disk, and available list high, convert more real storage to expanded storage Evaluate real storage peak period “available list” convert to expanded storage SET MDC XSTORE 0 0 (MDC in expanded storage has little value)
40	z/VM System Parameters SET SHARE Use RELATIVE 100 for single virtual CPU Use RELATIVE 200 for two virtual CPU SET SRM STORBUF – allow overcommit SET SRM STORBUF 300 300 300 SET SRM LDUBUF 100 80 60 SET QUICKDSP Use for only absolutely critical servers
41	Performance Resources