cover
TO: Mayor Richard C. Irvin
FROM: Michael Doerzaph, Commander Support Services
Michael R. Pegues, Chief Information Officer
DATE: September 16, 2019
SUBJECT:
Resolution authorizing the purchase of security system hardware, software, installation and annual maintenance services from CDW-G Corporation of Vernon Hills, IL under a National IPA purchasing agreement for an amount not to exceed $1,825,209.
PURPOSE:
This purchase continues the City’s multi-year plan to move toward a consolidated security platform citywide and seeks to resolve issues with certain portions of the City’s aging security infrastructure. This project is supported by multiple Departmental stakeholders including the Fire, Police, Public Works, Finance and Development Services. In addition, the City’s Chief Management Officer has also indicated his support for this project.
For many years, individual City Departments and Divisions were left to select and purchase their own security systems as needs arose and as new buildings and/or sites were brought on-line in the City. As a result, the City acquired a wide variety of both card access and camera security systems with varying degrees of functionality, support and upgrades over the years. Over the past few years, efforts have been underway in the City to consolidate these various facilities’ security systems under a single Genetec security platform.
By standardizing on a single platform, the City gains the following benefits:
City staff including both end user and support staff only need to learn and maintain one system,
The City can leverage volume pricing levels through economies of scale for hardware, software licensing and services,
The Police Department can utilize a single system to access all City cameras for surveillance and investigative purposes,
Video storage can be managed and maintained more efficiently and effectively including both shared storage and standardized retention periods based on clearly delineated City policies,
Access control and video camera access can be combined onto a single platform for ease of administration and reduced costs,
Enterprise application features can be made available to facilities and staff citywide at a reasonable cost and the City can avoid “one-off” low quality systems being installed in random facilities.
Several more years will likely be required to complete this process based on the cost and complexity of this project and the fact that some legacy systems are newer and more capable than others. Existing systems will be phased out over time as they age out and as funds are available.
An effort has been underway to catalog the major City facilities that are either in need of security systems for the first time or already have security systems in place but which have a failing or failed system that is in need of remediation or replacement. A prioritization of these facilities has been done and continues to be refined based on this cataloging effort and an assessment of the current state of the facilities’ respective security systems.
Among the most pressing City facilities identified for remediation and/or upgrades are the following:
Aurora Police headquarters campus,
Water & Sewer Maintenance Facility,
Development Services Center
Combined Fire Station 8 - Call Center - Community Room facility,
Route 59 Train Station*,
Route 25 Transportation Center*,
RiverEdge Park*
* Denotes sites where remediation has already begun based on an emergency basis (at an approved cost of $168,000 via City Council Resolution). (See Attached Resolution 18-326.)
In addition to replacing and upgrading security systems, the City is also in need of an annual city wide maintenance contract with a qualified security vendor. This contract would provide the City access to a vendor with security expertise in troubleshooting card access and camera issues at both the server level and the end device level. The vendor would provide preventative maintenance efforts including re-aiming cameras, cleaning camera lenses, and fixing/replacing inoperable devices. The City has experienced significant issues from the lack of a comprehensive maintenance contract with a qualified security vendor. This methodology of service is consistent with other complex systems such as the City’s HVAC systems in all buildings.
BACKGROUND:
The following are some of the highlights regarding the various City facilities experiencing issues with either existing security systems or a lack of security.
APD HQ Campus
The operations of the Police Headquarters facility requires functionality, flexibility and reliability that will support 24/7 mission critical life safety operations. Functionality supports movement and monitoring for detainees and facility security, flexibility is required for cost efficient use of staff, and reliability is needed to minimize the impact of system failures. The efficient and secure movement of staff and arrestees within the confines of the secure perimeter of the facility contributes to staff alertness, reduces staff fatigue, and increases the level of security. It is important that a security system is reliable. In a law enforcement facility, a failure can cause injury or even death in addition to costs associated with regaining control and establishment of order. All these factors contribute to the requirement of a highly engineered and reliable system in a law enforcement facility.
The electronic security system in the City of Aurora HQ consists of following subsystems:
1. Detention Security System (door control & monitoring and intercom & paging systems)
2. Access Control System (card readers)
3. Video Management and Recording System
4. Interview Recording and Monitoring System (Note this system was replaced in 2016.)
The Aurora Police Department’s current security systems were installed when the building was constructed. As a result the majority of the systems were installed in 2008-2009. While some software updates have been performed and some select storage and server components have been replaced or upgraded, the majority of the hardware and software has largely remained in place. The original system was purchased with a two (2) year system warranty and annual maintenance contract from the installing vendor, however, that warranty expired on October 16th, 2011. At the expiration of the original maintenance contract the Police Department opted not to renew the contract. As a result, the system has been under a “time and materials break fix” model of support for the past seven (7) years. This makes the majority of the systems components both software versions and hardware between seven (7) and ten (10) years old.
Following a series of system issues with the existing APD security systems, Latta Technical Services, Inc. out of Plano, Texas was commissioned to perform an assessment of the City’s existing security systems. This assessment produced a document titled “Security System Analysis of the City of Aurora Police HQ” dated December 29, 2015. The main goal of this report was to identify present status of the existing electronic security system and to provide the City of Aurora PD with the guideline on how to bring the system to a current industry standards. Latta Technical Services, Inc. (aka - Latta Tech) sought to gain a further understanding of the assets of the existing security systems as they related to physical equipment, components, software, cabling, and functionality.
The evaluation of existing systems required the review of information from all sources available and related to the installed work.
Among the findings of the 2015 Latta Tech survey are the following:
The existing ACS OnGuard software is outdate. The software version currently used is 6.3.249. This software version reached the end of life (EOL) in August 2012.
HID card readers I Class RP40 reached their EOL in Mar 2014.
HID Network Controllers reached their EOL in June 2016.
The Access Control System deficiencies track record list covers period December 2009 - December 2014. Up to that date, there were approximately 50 identified problems of which 39% of them were after the warranty period. The system malfunctioning was caused by various reasons including hardware failures and communication issues. The access control system is in need of upgrade and this should be very high on the system retrofit priority list.
The existing VMS NetDVMS software is outdate. This version of software has been superseded by OnSSI software Ocularis. The earliest versions of Occularis software are terminated so there is no technical support and no service releases and bug fixes.
Most of the existing Samsung cameras (SND-560, SNC-570, SND-460 and SNP-3300) are discontinued from production. (Since 2015, all of the cameras have been discontinued from production.)
The Video Management and Recording System deficiencies track record list covers period December 2009 - December 2014. There are approximately 70 identified problems of which 52% of them were after the warranty period. The system malfunctioning was caused by various reasons but most of them have been related to servers inability to manage the cameras. The OnSSi servers were replaced in 2014.
Security System Network (SSN) and Infrastructure does not provide any redundancy. The SSN network topology does not meet industry common practice for network redundancy and uninterruptible data exchange between all network nodes.
Note: The meaning of “end of life” for certain products depends on the product manufacturer but in most cases common denominators are:
Product no longer available for sale
No updates, no upgrades, no software packs
No or very limited technical support
Repair, replacement, substitution, or commercial credit depending on availability
The conclusion of the Latta Tech evaluation was that the biggest concern for the existing electronic security system in the City of Aurora PD HQ is future maintenance of the system.
In addition, a number of the system components have reached their end of life phase so future operation and maintenance will be extremely challenging. All software components that have reached EOL should be upgraded with the latest version of the same software or replaced with a new one.
One side effect of not maintaining an annual support contract and keeping the software versions up to date is that the legacy security camera software version no longer supports newer camera models. As the original cameras have begun to fail it is has become increasingly difficult to locate and purchase cameras of a vintage that are supported by the software. In addition, older less capable camera technologies are being purchased just to maintain an operational system.
The City of Aurora decided in 2015 that it would be standardizing and consolidating all City owned/managed facilities with access control to the Genetec platform as systems are upgraded and replaced. Considering that existing HID network controllers reached their EOL in June 2016 and that existing card readers are already outdated, it was decided that replacement of the existing access control system with the new Genetec platform was the recommended option.
The LattaTech assessment indicated that access control system replacement would have the following long term benefits for the system owner and users:
Common database for all city employees (same credentials for all city employees)
Common spare parts stock for all city buildings
Common knowledge base required for the system operation and maintenance
Competitive pricing based on volume
It was further determined that the existing VMS system hardware and software do not meet today’s industry standards for the video quality (resolution), length of recording and available system features.
Most new cameras used in the security industry today have:
Minimum resolution of 1080p which is 2.5 times better than any existing cameras in the facility.
HD format that allows better quality and coverage (16:9 ratio vs 4:3 for non HD)
State-of-the-art intelligent video analysis that reliably detects, tracks, and analyzes moving objects while suppressing unwanted alarms from spurious sources in the image.
Recording capacity for new law enforcement facilities is minimum of 90 days. All cameras should be continuously recorded at highest resolution, 15 images per second. Quiet time recording (no motion) shall be 4CIF, 2 images per second for all cameras.
The Genetec platform could be used for both video and access control system in the future. Switching to a single platform from the same manufacturer (Genetec) for both access control and video management systems would reduce number of issues related to communication between these two systems. It would also further support the benefits identified above for the following:
Common database for all city employees to access the system
Common spare parts stock for all city buildings
Common knowledge base required for the system operation and maintenance
Competitive pricing based on volume
A final critical recommendation was to provide Preventive Maintenance for the entire system that will consist of visual inspections, concise operational testing, adjustments, cleaning, lubrication, system backups, and performance verification of applications software and system electronics.
The Police Department continues to experience security related issues as the equipment continues to age and suffers from the lack of a preventative maintenance contract.
Water & Sewer Facility
The Water & Sewer facility has been identified as a City facility in need of additional security measures. The site is used to store a large number of high value equipment and inventory. Presently the site has very limited security and is in need of a major security overhaul. This facility is being targeted for gate control, access control and security cameras.
New Development Services Center
During the remodeling of the former PNC Bank facility in preparation for moving City staff into the newly named Development Services Center it was identified that the building would require both security access control and video surveillance cameras. It was determined that this facility would be brought onto the existing Genetec platform when time and funds were allocated. In preparation for this the facility was wired for said systems during the remodel.
The above facilities represent just some of the City facilities in need of remediation. The following facilities have been identified as high priority and among the first facilities to receive updates to their security.
Combined Fire Station 8 - Call Center - Community Room Facility
The combined Fire Station 8 - Call Center - Community Room facility provides critical services for the City including the City’s Fire Station 8, 311 Information Center and the City’s back up 9-1-1 Dispatch Center. Following an incident at the Fire Station 8 facility in Dec. 2017, it was discovered that the station’s security system was no longer operational due to a DVR system failure. This system was originally installed in the Station during the re-build of the facility in 2006-2007 making the hardware and software over eleven years old. Rather than try to repair an 11 year old system, it was decided that the system would be slated for replacement along with other failing systems in order to leverage a larger purchasing volume. The need to bring this site back on-line with operational security cameras was recently highlighted following a burglary of the Station.
Route 59 Train Station
The Route 59 Train Station is the busiest train station outside of the City of Chicago. Given the volume of traffic going through this station it is important that the City maintain adequate video surveillance.
The site’s cameras had been operating on two different recording systems. The Site’s parking lot cameras were replaced and migrated to the Genetec platform in 2017 during a re-design of the parking lot for the Station. In March 2019, the remaining cameras on the site were moved off a DVR system originally installed in 2005. This thirteen year old system had experienced a recent failure of one of its three DVR recorders.
Route 25 Transportation Center
While all of the cameras at the Route 25 Transportation Center have been migrated to the Genetec platform, many of the cameras are analog in nature and were originally installed in 2005. These cameras are failing at a higher rate based on their thirteen plus years of age and have an outdated level of video quality which hinders their intended effectiveness. Emergency repairs are currently underway at this site to address the most pressing needs, but an overall system upgrade will be required to the facility’s security systems.
RiverEdge Park
In the Spring of 2018, it was determined that RiverEdge Park had a number of cameras down around the site. It was also determined that the security room’s PC and monitor equipment was in need of a technology refresh base on the age and performance of the hardware. These issues are currently being remediated as part of the Ad hoc emergency SOW.
City Wide Maintenance Contract
A critical recommendation from LattaTech when evaluating the Police Headquarter’s security systems but which applies to any security system in the City was to provide annual Preventive Maintenance (PM) that will consist of visual inspections, concise operational testing, adjustments, cleaning, lubrication, system backups, and performance verification of applications software and system electronics. The repair of detected faulty components will be performed during the PM, otherwise these tasks will be performed during non-emergency corrective maintenance activities.
Annual Non-Emergency Corrective Maintenance (NECM) should be contracted and performed to correct any breakdowns or failures of the system or equipment whereby the failure does not cause risk to personnel or property, but may cause the system to operate in an inconvenient or degraded manner.
Emergency Corrective Maintenance (ECM) would be based on contract Service Level Agreements (SLAs) and would be performed within a pre-defined period of time to correct a breakdown or failure of the system or equipment whereby the failure causes undo risk to personnel or property, and to the point the system is inoperable.
The need for a preventative maintenance contract extends to all of the City’s facility security systems. The lack of a preventative maintenance contract has led to numerous system failures in the past which have hindered City operations and led to legal liability.
It should be noted that there will be an increase to the City’s annual recurring maintenance contract with Quantum Crossings for each site that is brought on-line and brought under the Quantum support contract. On average Quantum recommends budgeting approximately 20% of the total project cost (Services & Hardware) to estimate the annual recurring increase to the support contract. This figure is highly dependent on a multitude of variables including value of equipment being covered, complexity of accessing cameras and other security equipment, frequency of preventative maintenance requested by the City, etc.
See “Addendum 1” to this document for a breakdown of the above project costs (NOT including work already completed under the Emergency Ad Hoc repairs resolution).
Emergency Repairs
In addition to the above projects and a new annual maintenance contract, emergency repair work has been underway at a number of critical City properties in order to address immediate break/fix needs. Based on authorization from the City Council, City staff have been working with the Quantum to repair and replace dead cameras at a number of locations including the RiverEdge Park, Aurora Police Department Headquarters, Route 25 Train Station, and the Route 59 Train Station. These emergency repairs were deemed necessary based on the high liability these locations place on the City due to the level and nature of activities at these locations. The expenditures for these emergency repairs were covered under a previous Ad Hoc SOW and City Council Resolution in 2018.
DISCUSSION:
By undertaking this project, the City’s overall exposure to liability will be reduced and the community as a whole will benefit from the enhanced public safety that results from having a modern, comprehensive enterprise security solution. While the upfront costs to the City are significant, the risk of future litigation and resulting financial implications for putting this work off further cannot be understated.
IT seeks to leverage the National Intergovernmental Purchasing Alliance (National IPA) contract held by CDW-G as issued under an RFP by the City of Mesa, AZ. Under National IPA, Aurora participates in the cooperative buying power of more than 48,000 entities allowing the City to pay less for products and reduce administrative costs, which helps the City do more with less. National IPA is effective at creating more economical buying opportunities because they offer strength in numbers and years of government purchasing experience. Contracts are competitively solicited, evaluated and awarded by a government entity acting as the lead agency. Contracts are awarded utilizing industry best practices, processes and procedures, ensuring maximum value and absolute security with complete transparency of the process. National IPA is one of the most experienced cooperative team in public procurement, cooperative purchasing and supply chain management and a “Founding Member of National Coalition for Public Procurement”. National IPA offers a robust portfolio of quality contracts. National IPA partners are nationally-recognized, industry leading suppliers and manufacturers.
Lead agencies are considered to be among the best in public procurement and adhere to the highest standards of procurement. Past lead agencies have been the recipients of the 2017 Achievement of Excellence in Procurement (AEP) Award.
A Capital Improvement Project (C122) was requested and approved in 2018 for the 2019 budget in order to begin to address the Police Department’s security system issues. Due to budget constraints, however, the CIP was not fully funded at a level necessary to address all the concerns at the Police Department and other City facilities. As a result, IT worked with the Chief Finance Officer to identify additional funding and spread the project across multiple years. As a result, this purchase is being funded via approved 2019 and 2020 Proposed CIP funds and the Insurance Fund. The Insurance Fund budget will cover any amount over the $450,000 allocated annually in the CIP. The Insurance Fund will then be repaid from the Capital Improvements Fund.
FUNDING SOURCES FOR TOTAL PROJECT
Maintenance - Funded from the General Fund/IT Budget and Water Fund for those specific Facilities
2019 - $35,000/month - General Fund/IT/Water Funds
2020 - $425,000 - General Fund/IT/Water Funds
Capital Equipment and Installation - $1,407,000
2019 - Capital Improvements Plan - $450,000
2020 - Capital Improvements Plan - $450,000
2020 - Property and Casualty Insurance Fund - $507,000
2021 - Repayment of Insurance Fund by CIP - $507,000
IMPACT STATEMENT:
This project directly and indirectly impacts the general public, multiple City Departments and Divisions and outside entities. The direct impact to the general public is in the area of enhanced public safety and improved government services. Multiple City Departments and Divisions are impacted in both their operational activities, procedures, and staffing. In particular, the Police Department, Fire Department, Innovation & Core Services Department and Public Works Departments are directly affected by increased operational efficiencies, improved system capabilities, and streamlined procedures. Each of these Departments is aware of the project and have varying degrees of involvement with the planning and deployment of the project in their respective areas.
RECOMMENDATIONS:
Authorize the purchase of security system hardware, software and services from CDW-G Corporation of Vernon Hills, IL under a National IPA purchasing contract for an amount not to exceed $1,400,209. Authorize the implementation of an annual maintenance contract with Quantum for $425,000 for all Genetec security related infrastructure in the City of Aurora.
cc: Finance Committee
CITY OF AURORA, ILLINOIS
RESOLUTION NO. _________
DATE OF PASSAGE ________________
title
A Resolution authorizing the purchase of security system hardware, software, installation and annual maintenance services from CDW-G Corporation of Vernon Hills, IL under a National IPA purchasing agreement for an amount not to exceed $1,825,209.
body
WHEREAS, the City of Aurora has a population of more than 25,000 persons and is, therefore, a home rule unit under subsection (a) of Section 6 of Article VII of the Illinois Constitution of 1970; and
WHEREAS, subject to said Section, a home rule unit may exercise any power and perform any function pertaining to its government and affairs for the protection of the public health, safety, morals, and welfare; and
WHEREAS, the City has a mandate to provide for the safety and security of its facilities and the public at large, and:
WHEREAS, the City had a number of sites where security systems and cameras were in need of upgrades and repairs, and
WHEREAS, CDW-G offers security equipment and services under a National Intergovernmental Purchasing Alliance (National IPA) held by CDW-G as issued under an RFP by the City of Mesa, AZ and
WHEREAS, Aurora participates in the cooperative buying power of NIPA and the City chooses to leverage this contract for its own security purchases, and
WHEREAS, this request is being submitted as an joint purchase under the Mesa, AZ RFP and National IPA contract, and
WHEREAS, pursuant to Aurora Code Section 2-335(a)(3) the desired product is provided by only one source in this situation;
WHEREAS, the following fund’s accounts contain sufficient monies to cover this purchase: 101-1380-419.38-05 (General Fund), 255-1380-419.64-10 (SHAPE Fund), 255-1380-419.64-11 (SHAPE Fund), 340-1380-419.74-11 (Capital Improvement Fund), the Water Fund and the Property & Casualty Insurance Fund; and
NOW, THEREFORE, BE IT RESOLVED by the City Council of the City of Aurora, Illinois, as follows: authorize the purchase of security system hardware, software and services from CDW-G Corporation of Vernon Hills, IL under a National IPA purchasing contract for an amount not to exceed $1,825,209 and does hereby further authorize its Purchasing Director to issue the necessary orders to implement the same.